Module 3: Clean your data

Let’s imagine you work at
a board game manufacturer. As the quality
assurance manager, your job is to make sure every game is put
together correctly. One day you discover that the machine that produces
the games is having technical issues causing
misprinted cards and incorrect counts
of game pieces. Your manager asks you and the rest of the quality
assurance team to go through the affected
boxes and replace misprinted cards
and missing pieces. Your goal is to make
sure each box is usable. Searching through game
boxes for missing an incorrect game pieces
is similar to cleaning, joining, and validating
your data sets as part of exploratory data
analysis or EDA. Earlier, you learned how to
discover and structure data in order to understand and tell it stories in impactful ways. In this part of the
course, we will cover three of the other
practices of EDA: cleaning, joining, and validating data. Although there are
many different ways to clean join and validate data we will focus on
missing values and outliers, the need for transforming
categorical into numerical data and the
importance of input validation. Of course, we won’t just talk about the concepts of
cleaning, joining, and validating, you will learn how to apply them in
a Python notebook setting. We’ll use data sets that are comparable to what you
will see on the job. Along the way, we’ll go
through some important tips for improving your workplace
skills like communication. We will discuss when
to communicate with stakeholders and engineers about missing or outlier
values and about the ethical
implications you must consider when dealing with missing an outlier data values. These practices of EDA along with effective
communication and Python coding are
all essential to preparing a data set for the next steps in
the pace workflow. By following pace while performing the six
practices of EDA you will not only save time and energy on future processes, but you will be
more effective in finding and telling
the data story. One of my first
jobs at Google was a quantitative analyst
on Google Translate. It was my responsibility to help identify areas that we can
improve to help users. I recall a project where I was overly eager to
dive into the data. I spent countless hours
identifying opportunities ranging from when a user first downloads the app
to their second, third, and 20th time
using the product. I was proud of my
findings and I developed a long presentation to
share with my executives. Now while my insights
were well-received, the team was reluctant to invest in any of my ideas
because there were so many and they weren’t sure
which ones to prioritize. I realized after that meeting
that I needed a plan. I’d spent so much time mining
for insights that I was suffering from what my manager
called analysis paralysis. Instead, what I needed
to do was find out what issues were affecting the most number of users
and the severity of it. By organizing my thoughts
using this strategy, I was then able to
hone my analysis into one specific issue and then recommend clear actions to help. Like your job as a quality assurance
manager that we imagined at the
beginning of this video. As a data professional, you may be responsible
for cleaning up data sets so that
they’re ready to use. You won’t always be sure what obstacles you may
face with cleaning data, so let’s get started with
some of the most common.

Understanding Missing Data and Its Challenges:

• Definition: Missing data refers to values that are not stored for a variable in a dataset, often represented as N/A, NaN, or blank. It’s distinct from a data point of zero.
• Prevalence and Impact: Missing data is common and can significantly impact analysis and conclusions if not handled properly.
• Reasons: Missing data can occur due to various reasons, including computer errors, data entry mistakes, survey non-responses, or values not applicable to certain cases.
• Ethical Considerations: Be mindful of potential biases and assumptions when dealing with missing data.

Identifying and Quantifying Missing Data:

• Visualization Techniques: Use histograms, box plots, and scatter plots to visualize missing values and their patterns.
• Statistical Measures: Calculate the percentage of missing values for each variable to assess the extent of missingness.

Handling Missing Data Strategies:

1. Request Data Fill-in:
   • Contact data owners or conduct follow-up surveys to retrieve missing values.
   • Ideal when feasible and data collection is recent.
2. Delete Missing Data:
   • Remove rows or columns with missing values.
   • Suitable for low percentages of missingness and when deletion doesn’t introduce bias.
3. Create Missing Data Category:
   • Assign a unique category (e.g., “Missing”) for missing values in categorical variables.
   • Preserves information about missingness.
4. Fill in Representative Values (Imputation):
   • Replace missing values with estimates based on existing data.
   • Common methods:
     • Mean imputation: Replace with mean of the variable.
     • Median imputation: Replace with median of the variable.
     • Forward filling: Replace with previous non-missing value.
     • Backward filling: Replace with next non-missing value.
     • Predictive modeling: Use regression or machine learning to predict missing values.

Choosing the Best Strategy:

• Consider the amount and pattern of missing data, the nature of variables, analysis goals, and ethical implications.
• No one-size-fits-all solution; experimentation and evaluation are often necessary.

Additional Tips:

• Communicate with stakeholders about missing data and its implications for analysis.
• Document your missing data handling approach for transparency and reproducibility.
• Consider advanced techniques like multiple imputation for complex scenarios.

Staying Informed:

• Keep up-to-date with best practices and emerging techniques for handling missing data.
• The field of missing data analysis is constantly evolving.

Data encoded as N/A, NaN, or a blank is defined as zero.

False

Data encoded as N/A, NaN, or a blank is defined as a value that is not stored for a variable in a dataset. This is different from a data point of zero, which may be a missing value or a legitimate data point.

Remember earlier when you imagined
yourself as a quality assurance manager at a company assembling a board game. When you discovered that the machine that
produces the games was having technical issues and
causing game boxes to have pieces missing. You and your company could have chosen
a different number of paths forward. One, you could have thrown
away the impacted game boxes. Two, you could have left them as is and
sold them at a discounted rate. Or three, and
the one you chose in the example, you rummage through the boxes and
corrected the mistakes. There was a plan, strategy, or
protocol for dealing with missing pieces. The same should be true for any missing
values you find in the data set. When you have missing entries in your data
set, you have to decide on a plan for dealing with them. Even more than that, you must be ready
to communicate with stakeholders if missing values impact your analysis. Missing data, which is often
encoded as N/A, NaN or blank, is defined as a value that is not
stored for a variable in a set of data. This is different from
a data point of zero. I’ll explain more on that in a bit. Data professionals often experience
the challenge of missing data. Every data set is different. So, there are a wide variety of
reasons for values to be missing. Everything from a computer
error during a data upload, to someone simply forgetting to input it. Depending on the number of
entries in your data set and the quantity of missing values. The impact of data fields
with a not a number or NaN’s can range from
negligible to substantial. The impact will also determine how
you communicate to stakeholders or clients, ranging from a note at
the bottom of a data visualization. About the fact that there’s missing data,
to a face to face meeting about how analysis cannot be completed due
to the large number of missing values. Here’s an example of the impact
missing data has on a data set and the ethical questions
missing data creates. Imagine you’re a data professional
trying to learn more about sleep habits. You email questionnaire to, say,
100 people and all of them completed. One of the questions asked, do you
sleep exactly eight hours each night? The answer choices are yes, no, or unsure. 9 people answered yes, 9 answered no,
and 15 answered unsure. Because only 33 people responded to
that question, it’s hard to draw a definitive conclusion about
the sleep habits of all 100 students. Since 100 people completed
the questionnaire and only 33 responded to this question, 67% of
the replies would be considered missing. These missing values greatly impact
the questions resulting data. It would be unwise for a data professional
to try to use the 33% response rate in order to draw conclusions
about the entire population. If you can encounter this level of
missing values while working as a data professional, you should communicate
the inability to complete any analysis and suggest possible solutions. Another challenge that can come up
during EDA, is with the value zero. In some data sets, a zero could
be considered a missing value. But in other data sets could
be a legitimate data point. In some data sets, a NaN or
blank space might be a mistake. Someone forgot to fill it in. Or the data may have been left
blank because the column may not apply to that data point. Data professionals unsure of whether or
not the blank spaces intentional, should ask a stakeholder or the owner of
the data to confirm the spaces legitimacy. Whenever you find missing data, you have
a choice to make on how to handle it. As a data professional, you should
consider how the missing data might impact stakeholders and
who should be made aware. Here are four common ways
to handle missing data. One, you requested the missing values
be filled in by the owner of the data. Two, you delete the missing columns,
rows, or values which would work best if the total
count of missing data is relatively low. Three, you create a NaN category. Or four, you derive new representative
values such as taking the median or average of the values that aren’t missing. First, we’ll talk about
filling in the missing data. If there are large quantities of
missing values in the data you receive, the best method to handle the missing data
would be to contact the owners of the data and request that that data be filled in. For example, in our sleep example,
you could send a follow up request for people to log a response to
that particular question. Or you could rephrase the question,
how many hours do you sleep each night? However, in many data sets, you see while
working as a data professional, the option to get missing data filled in may not
be feasible for a variety of reasons. For example, if you’re working on
a project that is time sensitive, you might not have time to gather
more data before your deadline. Or the data cannot be retrieved because
the study happened too long ago. If filling in missing
data isn’t an option, you can also choose the option
of deleting the missing data. Though some might assume it’s
not appropriate to delete data, removing rows or
whole columns of data is ideal. If there isn’t a large
percentage of NaN’s or the values are not going to impact
the business plan for the data. One thing to watch out for though, is discarding missing data
that is not missing at random. Deleting values that have been
left blank intentionally, can skew the results of your analysis. For example, in the sleep questionnaire,
if you were to remove all the people that left that question blank,
you delete a majority of the data. A third option,
is to make the NaN their own category. Which is a good strategy if the missing
data itself is categorical rather than numerical. For example, in the sleep questionnaire,
if you were to put all the non responses into their own
category called answer not recorded, you would be creating a category for
the missing data. Finally, there is the strategy of
filling in the missing data by creating a representative value. This strategy can be more useful
with business plans that call for a predicted value or forecast. There are multiple methods within this
filling in strategy to choose from. Including the four most common,
forward filling, backward filling, deriving mean values, and
deriving median values. We will talk about how to do all these
missing data operations using python in another video. These are the most common methods for
handling missing data. Choosing which method to use comes with
experience, intuition, and reasoning. Sometimes you’ll have the opportunity
to use more than one option with the data set. Every dataset and
project plan will be different. So, you will be making this determination
each time you encounter new data. Sometimes, you’ll want to confer
with peers, or managers, or stakeholders in order to make a decision. The impact missing values
has on your analysis, should be the determining factor
in whom you should contact. As a data professional, it is
essential that you are thoughtful and intentional about how
missing data are addressed. Consider the quantity of NaNs and their importance in relation
to the project plan. Ask yourself, how will this
approach impact this data set? And what are the ethical considerations? Just like the board game scenario, it is essential to determine
a strategy and decide on a plan.

Identifying and Analyzing Missing Data in NOAA Lightning Strike Data: A Tutorial

This tutorial explores the identification and analysis of missing data within two NOAA lightning strike datasets for August 2018. By working through these steps, you’ll gain valuable insight into handling missing data in your own projects.

Data Sets:

• df: Contains basic information like date, latitude, longitude, and number of strikes.
• df_zip: Extends df with additional details like zip code, city, state, and state code.

Identifying Missing Data:

1. Merging Dataframes: Combine df and df_zip using pandas’ merge function.
2. Locating Missing Values: Analyze the merged dataframe to identify missing entries in columns like “zip code,” “city,” “state,” and “state code” (likely absent for water bodies).
3. Quantifying Missing Data:
   • Use pd.isnull.all and the shape function to find the total number of missing entries (e.g., 393,830).
   • Utilize the info function to examine the “non-null count” for each column, confirming which ones have missing data.

Analyzing Missing Data:

1. Focus on Missing Values: Create a new dataframe, df_null_geo, containing only relevant columns and the missing values (e.g., latitude, longitude, “number of strikes y” with missing zip code data).
2. Visualization with Plotly Express:
   • Import the express package from plotly for efficient visualization.
   • Use plotly express scatter geo to create a map highlighting the locations of missing data points.
   • Filter the data to focus on areas with significant lightning strikes (e.g., > 300).
   • Size the data points based on the “number of strikes y” values.
   • Customize the plot with a title and appropriate geographic scope (e.g., geo_scope='usa').

Key Takeaways:

• Missing data in “zip code,” “city,” and “state code” columns likely occur for locations like water bodies, which lack these designations.
• Data cleaning and analysis can reveal hidden insights, such as the association between missing values and water locations.
• Optimizing visualizations with large datasets is crucial (e.g., using express package).

Further Exploration:

• Experiment with different data cleaning techniques to handle missing values based on your specific needs and analysis goals.
• Investigate the potential impact of missing data on your analysis and conclusions.
• Consider alternative visualization methods to effectively represent the distribution and characteristics of missing data.

By following these steps and applying them to your own datasets, you can gain a deeper understanding of missing data and its implications for your analysis, leading to more robust and reliable conclusions.

Remember, data cleaning and analysis are essential steps in any data-driven endeavor, and effectively handling missing data is key to drawing accurate and meaningful insights from your information.

What is indicated by the term null?

The data is missing.

The term null indicates that the data is missing.

Now that we reviewed the importance of
having a plan to address missing data. Let’s talk about how to identify and decide how to work with missing
data in a Python notebook. We’ll use the data you’re familiar with, the group of NOAA
lightning strike data sets. The goal of this video is to learn
how to identify missing data. You’ll be provided two different
slices of data from August 2018. One slice will have the columns date,
center point geom, latitude, longitude,
and number of strikes. The second slice of the data will include
the same columns as the first slice, along with the zip code,
state, city, and state code. You’ll learn through comparisons of these
two data sets, how to find missing data. Before we look for any missing values, we need to import some Python
libraries and packages. We’ll use pandas, NumPy, seaborn, datetime, and pyplot from matplotlib. We will begin by looking at only
the number of lightning strikes for August 2018. Using less data in a Python
notebook allows you to spend more time coding and
less time waiting for the code to run. For this exercise, our first data set will have two
additional columns called latitude and longitude which were points pulled
from the center point geom column. Let’s explore the column pandas and
the overall size of this first data set using the functions that you’re
already familiar with, head and shape. We’ll find the two additional
columns we just mentioned for latitude and longitude as well as
a shape of 717,530 rows and 5 columns. Let’s put this aside as our first
data frame saved as df.shape. Next, we’ll use the second data set for
August 2018. This one has columns for
zip code which is a postcode delivery area number in the US,
as well as city, state, and a column titled state code that has two
letter abbreviation of states in the US. We’ll call the second data frame df zip, so we don’t get confused with the first data set we created earlier. Let’s run the same two functions head and shape on df _zip. The head function returns what we expected out of the data set. The data columns, zip code, city, state, state code center point geom, and number of strikes. The shape function, however, returned 323,700 rows and 7 columns. Given that this data frame is also from August 2018, we expected to see the same number of rows as the first data frame, 717,530. To further explore this, let’s put the two data frames together with the merge function. We’ll need to create a new data frame. We’ll call it df _joined = df .merge. This df .merge indicates that we want to merge the first data frame we saved named df with another data frame. For the merge put df _zip in the argument field, which is the data frame we want to merge with df. Next, fill in the parameters for the merge, including how= ‘ left ‘ ,. Finally, input the last parameter on= ‘date’, center _point _geom ‘. These two parameters how and on, tell Python which way to join the data. When we run the head function on our new data frame we find NaNs listed for the columns zip code, city, state, state code, and number of strikes y. Now, that the data is merged, let’s use a basic function we learned in a previous lesson to search for missing data, describe. Input, df _joined.describe( ) and leave the argument field blank. We run the cell, we find that the total count of lightning strikes has been divided into two, number of strikes x and number of strikes y. Once we merge the two data frames, pandas automatically separated number of strikes into data entries with full data rows and data entries that have missing entries. We can find the total amount of data that’s missing using this next piece of code. We’ll create a data frame called df _null _geo by taking our joined data frame df _joined and then adding the pandas function pd.isnull all one word. The term null indicates that the data is missing. This function pulls all of the missing values from the df _joined data frame. We already know which of the columns have missing values. Since we’re interested in finding the total, we can use any one of those columns. For this code, we selected state_code. After that, we’ll use the shape function to give us the total rows and columns, which comes to 393,830 and 10 respectively. For even more detail, use the info function which we used in another video. When we input df _joined.info( ) with the argument field left blank. We get the column names and another column called non_null count. Non-null count is the total number of data entries for a data column that are not blank. A quick check of the non-null count column helps us confirm which columns have missing data, zip code, city, state, and state code. In fact, if we subtract the total number of strikes x and the number of strikes y the result is the number we identified earlier, 393,830. Let’s take a look at the top portion of the new data frame we created, df _null geo.head using
the head function. The output tells us exactly what
we expected to find in the data frame we created. In the top five lines of data,
the columns zip code, city, state, state code and number of strikes y have
the NaN values in the cells. Now that we have pinpointed exactly what
values are missing in our data frame. The last thing we’ll do in this notebook
is learn how these missing values impact our data. The best way to do that is to
create a data visualization. A plotted map will help us see where
the majority of the missing values are located geographically. To design the map, let’s start
by creating another data frame. This one we’ll call top_missing. With this data frame we
are gathering only the data coms we will need in order to plot
a geographic visualization, latitude, longitude, and
number of strikes x. As you’ll recall from
earlier in this video, number of strikes x
includes all of the 717,530 data rows while number of
strikes y are a segment of those 717,530 with missing data in the zip code, state, and state code columns. You’ll find that it’s helpful to group the
columns by latitude first then longitude. This will make it easier for
most data processors to plot. Lastly, we’ll sort the lightning
strikes x column by some of values. In the ascending field will input,
False, because we want the largest sums of lightning strikes
at the top of the data frame. Finally, let’s take a look at what we
built using the head function again. After running the cell,
the first 10 rows show the most strikes are falling on and
around the same latitudes and longitudes. Let me give you a helpful hint before
we plot a visualization using this data frame. You should import the express
package from plotly first. Express is a helpful package that
speeds up coding by doing a lot of the back-end work for you. If we don’t use express for this
particular data set which has hundreds of thousands of points to plot your
run cell times could be long or the code could even break. Now, it’s time to create the map. Our format for the graph will be from
plotly express called scatter geo. As indicated by its name,
this graph type is used for scatter plots on a geographic map. In the argument field will input the data
frame we just created, top missing. For the parameters in the argument field, we want to filter the number of
values to be only those latitudes and longitudes with lightning
strikes of more than 300. Naturally, we plug latitude and longitude
into their appropriate parameter spots. Then for the size parameter, we plug in
the final column, number of strikes x. Lastly, let’s use the update
layout function within the figure to give the plot title,
Missing data. Now, we create it. It’s a nice geographic visualization but
we really don’t need the global scale. Let’s scale it down to only the geographic
area that we are interested in, the United States. So, let’s copy the same
code we just wrote, this time though we add
the parameter geo_scope= ‘usa’ ,. This will limit the geographic scope
to just the United States of America. Then we can plot it again and
check the output. The resulting map shows a majority
of these missing values cropping up along the borders or
in spots over bodies of water like lakes, the ocean, or the Gulf of Mexico. Given that the missing data were
in the state abbreviations and zip code columns, it does make sense
why those data points were left blank. There are no zip codes for
bodies of water. You’ll also notice some other locations
with missing data that are not over bodies of water. These types of missing data,
latitude and longitude on land are the kind of missing data you would
want to reach out to the NOA A about. What’s interesting about cleaning
the lightning strike data and looking for missing data values is that we
learn something new in the process. We found one of the stories
hidden in the data. The data values were missing because
the lightning strikes were over water. It goes to show you that data
cleaning may sound tedious but you’ll never know what you learn.

Hi, my name is Remy. I’m a customer engineer
for Google Cloud. A customer engineer is
someone who focuses in some technical field, for me, it’s data analytics
and machine learning, and is the expert in
that field and helps their customer answer
questions and build things related to
data analytics. I had no exaggeration around 30 different jobs before I got into the field
that I’m currently in, I was a cashier, a hostess, a cocktail waitress,
an office assistant. Quite frankly, I wasn’t very
good at any of those jobs. If somethings like repetitive, I can’t concentrate
and then I end up making silly mistakes. That’s what’s so great
about data science is a lot of the minutiae
and repetitive tasks, software does it for you or you can write a script to do it. There is definitely
no typical day in the life of a
customer engineer. Usually I will start the
day by looking at emails. I’ll usually have some
questions from a customer and other colleague related to
data that I can help answer. There’s a lot of
informational emails about new features, products,
or techniques. I have to read those and learn about them because
it’s my job to be the person who
knows about all of these things for our customers. I’ll also typically have
a few meetings where I’m working with our account
and customer teams where we try to strategize how
we are going to help customers and get them to do
new things with our tools. A new feature that I’ve
been working with is related to our data
warehouse BigQuery. You can actually create
machine learning models within BigQuery so you don’t have to move the data anywhere. I have a variety
of customers that use BigQuery Machine Learning. So I can now come to
them and share with them this new feature and show them how
it actually works. If while they’re using it, they might have an
issue that comes up, they get some error, I can help troubleshoot
that problem for them. There’s so much going on in this field of data
and it’s also new. It’s going to feel overwhelming. I’ve been doing it for
almost a decade and I still get overwhelmed
on a daily basis. The best thing you can
do is pick a software, pick a language, maybe
a specific area, and just focus on that. Get really good at one
thing and then you can expand from there.

Here’s a tutorial on outliers in data analysis:

Outliers: Understanding the Unexpected

What are Outliers?

• Outliers are data points that significantly differ from the other observations in a dataset, often deviating from the overall pattern or trend.
• They can arise due to various reasons, including:
  • Measurement errors or data entry mistakes
  • Natural variations within a population
  • True outliers that represent unique or exceptional cases

Types of Outliers:

1. Global Outliers:
   • Values that are far away from the overall distribution of the data.
   • Often considered errors or anomalies and may be removed for analysis.
2. Contextual Outliers:
   • Values that are unusual within a specific context or condition.
   • They might be valid data points but need careful interpretation.
3. Collective Outliers:
   • Groups of data points that deviate together from the rest of the dataset.
   • May indicate a distinct subgroup or a different underlying process.

Why Outliers Matter:

• Outliers can significantly impact statistical analyses and model building.
• They can distort measures of central tendency (mean, median) and variability (standard deviation, range).
• They can influence model fitting and prediction accuracy, leading to misleading results.

Detecting Outliers:

• Visualization Techniques:
  • Box plots
  • Scatter plots
  • Histograms
• Statistical Methods:
  • Z-scores
  • IQR (Interquartile Range) method

Handling Outliers:

• Removal:
  • Consider removing outliers if they are clearly errors or have a strong negative impact on analysis.
  • Use caution and transparency when removing data points.
• Transformation:
  • Apply mathematical transformations (e.g., log transformation) to reduce the impact of outliers.
• Modeling:
  • Use robust statistical methods that are less sensitive to outliers.

Ethical Considerations:

• Carefully consider the reasons behind outliers and their implications before removing them.
• Avoid removing outliers simply to improve results or fit a desired model.
• Document any outlier removal or treatment decisions for transparency.

Remember:

• Outliers can provide valuable insights into data patterns and underlying processes.
• Treat them with caution and consider their context before making decisions.
• Employ appropriate methods to detect, analyze, and handle outliers ethically to ensure accurate and reliable data analysis.

Fill in the blank: Outliers are observations that are an _____ distance from other values.

abnormal

Outliers are observations that are an abnormal distance from other values. They may also be observations that are abnormal compared to the overall pattern of the data population.

As a child, did you ever play a puzzle game
in which you are supposed to look
at a picture and find an object that
doesn’t belong? Maybe sometimes the
object was obvious, but other times, it may have been more difficult. For example, if you’re
given a picture of a bustling marketplace full of people selling
fruits, vegetables, and grains, it probably
takes some time to find the item for sale that’s
not in the right place. Similar to visiting
a busy market. When you encounter a dataset
that hasn’t been cleaned, it can be really challenging. Data may at first
seem structured, but it often can be hard
to determine whether there are any data points that are measurably different
from others. These extreme data observations, the ones that stand
out from others, are known as outliers. Outliers are observations that are an abnormal distance from other values or an overall
pattern in a data population. In this video, we will discuss the three main types
of outliers and why it is important for
data professionals to identify them in the
data we analyze. As a data professional, you should be fully aware
of the beginnings and ends, highs and lows, and extreme points of your data across every variable
in the dataset. These values will often
be your outliers. It is essential that
you recognize them, what their value is,
and where they are. Otherwise, they will likely skew any conclusions you draw or
models you build from them. There are three different
types of outliers we will discuss in
this video: global, contextual, and
collective outliers. Later, you’ll learn how to find outliers and work
with them in Python. The first outliers will discuss tend to be the easiest
data points to detect. Global outliers are values that are completely
different from the overall data group and have no association with
any other outliers. They may be inaccuracies,
typographical errors, or just extreme values you typically don’t
see in a dataset. For example, if we had
a set of human heights, 1.7 meters, 1.9
meters, 1.6 meters, 1.8 meters, 7.9 meters, 1.8 meters, or 1.7 meters, the outlier is fairly obvious. Typically, global
outliers should be thrown out to create
a predictive model. Contextual outliers can
be trickier to spot. Contextual outliers
are normal data points under certain conditions but become anomalies under
most other conditions. As an example, movie
sales are expected to be much larger when a
film is first released. If there is a huge spike
in sales a decade later, that would typically
be considered abnormal or a
contextual outlier. These outliers are more
common in time-series data. Another example might
be an outlier only in a specific single
category of data. For instance, 2.5 meters may be a normal enough size for a category called
mammal heights, but if the mammal is
found out to be a mouse, we would likely consider
this an outlier despite it fitting in with
other mammal heights. Lastly, we have
collective outliers. Collective outliers are a group of abnormal points that follow similar patterns and are isolated from the rest
of the population. Think of a parking
lot at a store. It is not uncommon to
have cars or scooters coming and leaving consistently during the hours
a store is open, but to have a full parking lot after the store is closed would be considered a
collective outlier. It could be that there is a company party or a
local event nearby, which would explain
the outlier of cars parked in the store
parking lot after hours. One useful way to find these different types
of outliers in our data is something we’ve
already discussed quite a bit, visualization. It’ll be easier to see any
big dips or giant blips in the data when we plot it on a line graph or a bar chart. No matter how you discover that your data contains
global, contextual, or collective outliers. It is essential that your EDA includes a strategy
for dealing with them. It will be up to you to decide
how outliers need to be represented or whether they need to be removed completely. The decision on what to
do must always be done in the context to the dataset and the business
plan for the data. As mentioned in other videos, always consider the
ethical implications of any decision you
make about outliers. When you’re working as
a data professional, it may be tempting to remove
outliers to improve results, predictions or forecasts. But you will need
to ask yourself, does that change the data story? For example, let’s
imagine that you are a data professional
at a retail business. You review two years of sales data and find
that there’s one month where sales of a
typically popular item are down dramatically. You might at first
assume that there’s a typo in the reported data, but as a diligent
data professional, you ask questions about it. You discover that
several top salespeople were on leave during
that month and your company advertised
a new product which slowed sales on
the existing item. It is likely that both of these issues led to
a drop in sales. Rather than dismissing the
drop in sale as an outlier, this information will be
helpful for the business team. Using this information,
managers can prevent a future drop in
sales numbers ahead of the product launches by hiring additional
sales team members or limiting requested
employee leave during those critical times. Later, you’ll learn how to find outliers and work
with them in Python.

⚡ Striking Down Outliers: Exploring Anomalies in Lightning Strike Data

Welcome data detectives! Today, we’ll embark on a thrilling investigation: finding outliers in US lightning strike data spanning 33 years. Prepare to uncover hidden patterns, understand data distributions, and wield powerful Python tools to expose these statistical anomalies.

The Case of the Mysterious Strikes:

Our dataset holds the total lightning strikes recorded in the US from 1987 to 2020. But can we trust every data point? Enter outliers: unusual data points that deviate significantly from the majority. Identifying them is crucial for accurate analysis and avoiding misleading conclusions.

Our Toolkit for Unmasking Outliers:

• Python Libraries: We’ll use Pandas, NumPy, Seaborn, and Matplotlib for data manipulation, analysis, and visualization.
• Statistical Measures: Mean, median, interquartile range (IQR), quantiles – these statistics will unveil data distribution trends and potential outliers.
• Boxplots and Scatterplots: These visual powerhouses will showcase data distribution and highlight outliers for closer inspection.

The Investigation Begins:

1. Importing the Data and Libraries: Start by importing our trusty libraries and loading the lightning strike data into a Pandas DataFrame.
2. Data Wrangling: Let’s make the data easier to read and analyze. Use functions like .head() to preview the data and consider scaling large numbers for better visualization.
3. Mean and Median: Unveiling the Center: Calculate the mean and median of the total lightning strikes. Are they close or far apart? This hints at the data distribution (normal, skewed, etc.).
4. Boxplot: Unveiling the Outliers: Generate a boxplot to visualize the data distribution. Look for data points beyond the whiskers – those are potential outliers!
5. Interquartile Range and Quantiles: Calculate the IQR and identify the 25th and 75th percentiles (quartiles). This helps define the “normal” range of data.
6. Outlier Thresholds: Set upper and lower limits based on 1.5 times the IQR. Any data point beyond these limits is considered an outlier.
7. Identifying the Culprits: Use logical indexing to isolate data points below the lower limit – our identified outliers!
8. Scatterplot: Seeing Outliers in Action: Create a scatterplot with year as the X-axis and number of strikes as the Y-axis. Color outliers for easy identification.
9. Investigating Individual Outliers: Zoom in on specific outliers (e.g., 1987 and 2019). Check for missing data, data entry errors, or unusual events that might explain the anomaly.
10. Cleaning the Data: Decide how to handle outliers. Exclude them if they significantly skew the analysis, or adjust values if justified.
11. Recalculating Statistics: Recalculate the mean and median excluding outliers. Observe the difference in data distribution.

Conclusion:

By exploring statistical measures, visualizations, and investigating individual cases, you’ve successfully identified and analyzed outliers in your lightning strike data. Remember, understanding outliers is critical for drawing accurate conclusions and making informed decisions based on your data.

Bonus:

• Explore different outlier detection algorithms like Z-score or Median Absolute Deviation (MAD).
• Learn about outlier treatment methods like winsorization or data transformation.
• Apply these techniques to analyze other datasets and become a master of data anomaly detection!

So, grab your Python tools, put on your data detective hat, and get ready to strike down those outliers!

Docstrings are useful within a line of Python code, but they cannot be exported to create library documentation.

False

Docstrings are lines of text following a method or function that explain to others what that method or function does. They can also be easily exported to create library documentation.

Earlier, you learned about
three types of outliers, global, contextual,
and collective. Now that we discussed outliers
on a conceptual level, I’ll show you how
to identify them and analyze their impact
in a Python notebook. Our goal will be to
identify outliers across a 33-year span of total lightning strike
counts in the United States. As usual let’s start by importing our libraries
and packages, pandas, numpy, seaborne,
and pyplot from matplotlib. We’ll continue using our NOAA
lightening strike dataset. This time we’ll group
the total sum of the lightning strikes in the United States by
year from 1987-2020. Let’s first reveal
the top 10 rows of data using the head function. The dataset has two
numeric columns, year and number of strikes. As you’ll notice,
we’re dealing with some fairly large numbers for the lightening
strike totals. It would be helpful
to make them a bit shorter and easier to
read in visualizations. To do that, let’s write a
readable numbers function. Below the function, there is a long explanatory text inside a triple set
of quotation marks. This is called a documentation
string or docstring. A documentation string
or docstring is a line of text
following a method or function that is used to
explain to others using your code what this
method or function does. A docstring represents good documentation
practices in Python. It makes the code easier
to understand and can be easily exported to create
library documentation. A docstring was already provided for this readable
numbers function. Next we’ll form an
if else statement. We want to code any number above six digits to be formatted
to one decimal place, followed by an M. Under
the else statement we are formatting numbers with
more than three digits to the same one decimal place, followed by the letter K.
Under these statements, we define the new
DataFrame we want titled number of
strikes readable. We then use the apply function to set our readable numbers statement to only the number of strikes column
in the DataFrame. Using the head function, we’ll see the output
of our newest code. The new column has readable
values of 15.6 million, 209,000 and 44.6 million
in its first three rows, where before they were just long strings
of numeric digits. This indicates that we coded the read numbers
function correctly. Now that we have more
readable values, let’s plot our data. As you’ll recall, our goal is
to find any outliers among the total number of
lightning strikes in 33 years of NOAA data. One way to find any
outliers would be to investigate the mean and
median of this data. The mean is the
average number of the given values and the median is the middle point
of the given values. When we code for
these two values, we use the numpy functions
of mean and median. Were using the readable
numbers for simplicity. The resulting
output is the mean, 26.8 million and the
median 28.3 million. With the mean being two million strikes less than the median, we suspect the data distribution is likely skewed to the left. The left side of the distribution will be a
good place to investigate. One effective way to visualize
outliers is a boxplot. As we discussed in
a previous video, a boxplot divides
the distribution of data points into four main
quadrants or quartiles. The boxplot is helpful
in visualizing and confirming the outliers
we’ve already found. Let’s plot the boxplot first and then we’ll talk
about it in more detail. To design a boxplot
using Python, we’ll use the seaborne
boxplot function. We’ll use the number
of strikes column from our DataFrame as the
basis for the plot. Next, we’ll use
set x tick labels based on the readable
numbers statement we created already. X tick labels simply
allow us to give names to the x-axis marks so that they are
read more easily. Lastly, we input the x and
y labels, and the title. You’ll recall that the purpose
of a boxplot is to show the distribution of values separated into
quadrants or quartiles. What we’re most focused
on with our plot are that the two points beyond the far left of the line
are below 10 million. This data visualization
is showing us very clearly there are outliers included in our group
of data points. The blue rectangles
in your boxplot are called the interquartile range. This range represents
the difference between the third quartile and the first quartile of
a set of data values. A standard rule statistics which you will learn more
about another course is that any data point
that falls beyond 1.5 times the blue boxes
are considered outliers. This next piece
of code will help us define what value is at 1.5 times below our
interquartile range. We’ll use the quantile function
to define what we’ll call percentile 25 and percentile 75. We enter our data column, number of strikes
for our DataFrame, and then 0.25, and 0.75 in the argument fields. The values that occur between the 75th and 25th percentile
are the interquartile range. We create a statement for
interquartile range, IQR, defining percentile 75 minus percentile 25 is equal to IQR. Now, let’s define
two statements; upper limit and lower limit. Upper limit is equal to
percentile 75 plus 1.5 times IQR. Lower limit is percentile
25 minus 1.5 times the IQR. Lastly, let’s have Python provide the exact value
of the lower limit. We’ll input print,
followed by the text. Lower limit is colon plus
sign readable numbers, and lower limit in
the argument fields. After running the cell, we learn that the lower
limit is 8.6 million. Don’t worry if
you’re still trying to understand
interquartile range. This concept will be explored further in depth
later in the program. Next, let’s plot
the values that lie below our lower limit
of 8.6 million. Place the number of strikes column inside the
brackets next to df. Then use the less than symbol followed by the lower
limit we just derived. Now here are the outliers. One great way of seeing these
outliers in relation to the rest of the data points
is a data visualization. For this plot, let’s
do a scatter plot. You may recall that a scatter plot represents
relationships between different variables with
individual data points without a connecting line. To start the scatter plot, let’s first add labels to
each point on the plot. We’ll define them using the add labels function
for x and y points. The plt.text function allows us to define how we want the text of each data
point to appear. In this case, we tell the
texts to start 0.05 pixels to the left of the data
point and just start 0.5 pixels above the datapoint. We also defined the
number of values using our readable number statement from the beginning of the video. Next, let’s add color
to the data points. Let’s make the outlier points red and the other points blue. To do that, we’ll define
colors by the lower limit. For those that fall below our previously
defined lower limit, we’ll code that they
should be r for red, and everything else will
code as b for blue. Our next line of code
should feel familiar. We start by configuring
our visualization size. In this case, we code
16 and eight inches, which are the default
units for this function. Next is our code for
scatter plots, ax.scatter. The ax refers to axis, which tells the notebook to plot the data on an x
versus y data graphic. Inside the argument field, we first input the x-axis
and then the y-axis, which is year and number
of strikes, respectively. For the rest of the parameters, we fill in the x and y
label and the title. We finish the parameters
of our scatter plot by defining the x axis
tick labels and setting the rotation
of the x-axis ticks to a rotation
of 45 degrees. The resulting plot shows
us the years 1987 and 2019 are the two values
in red or two outliers. Now that we’ve narrowed our
scope of the outliers to just two years,1987 and 2019, we can do a little
more investigating. Let’s start with the 2019
lightning strike data. If we import just the 2019 data, the first thing
we’ll want to do, as we learned in a
previous video is convert our date
column to date-time. Next, we’ll create
some new columns in our dataframe month
and month text. You’ll notice for the date, we use the str.slice function to cut the month names down to
just the first three letters. Finally, we create
a dataframe that groups the total number of lightning strikes
for each month. The result explains why
the year is an outlier. For the 2019 dataset only lightning strikes from the month of December
had been inputted. If you were a data
professional tasked with calculating lightening
strike risk, you would first research the NOAA documentation available to see if there’s
a reason given. If you don’t find anything
based on your research, you would then ask the
NOAA why lightning strikes haven’t been recorded for
the other 11 months of 2019. When we do the same
code from 1987, the resulting output is
different from 2019. We find there are
lightning strikes recorded for each month in 1987. This difference that 2019 does not have data
for all months, and that 1987 does have
data for all months, helps us to know how to
handle these two outliers. For 2019, it would make sense to exclude it from our analysis. As for 1987, we
recognize that it is indeed an outlier
but it should not be excluded from our analysis
because it does have lightening strike
totals included for each month of the year. Now, before we end,
let’s return to the mean and median values
that we calculated earlier. First, let’s create
a new data-frame, including only data points that exclude the outliers
we identified. Our goal here is to
show the mean and median of the dataset
excluding the outliers. To do this, we’ll
use a data-frame without outliers we just created and our readable numbers
statement to only include data points that are greater than our lower limit. When we exclude those outliers, the mean and median are
much closer together, suggesting a fairly evenly
distributed dataset. Now, let’s revisit
the goal we set earlier: to identify outliers across the 33-year span of total lightning strike
counts in the United States. We discussed a lot of new concept and we
achieved this goal.

Here’s a tutorial on Categorical Data and Encoding Techniques:

Understanding Categorical Data:

• Definition: Data that represents qualitative descriptions rather than numerical values.
• Examples: Gender (male/female), occupation (doctor, teacher, engineer), education level (high school, bachelor’s, master’s)
• Importance: Often used to capture descriptive characteristics in various datasets.

Challenges with Categorical Data:

• Numerical Preference: Many data models and algorithms are designed to work primarily with numerical data.
• Conversion Necessity: Categorical data often needs to be converted into numerical representations for effective analysis and modeling.

Encoding Techniques:

1. Dummy Variables (One-Hot Encoding):
   • Create new binary variables (0 or 1) for each unique category.
   • Each observation has a “1” in the column representing its category and “0” in all other category columns.
   • Example: Gender (male/female) becomes two columns: “Male” (1 for males, 0 for females) and “Female” (1 for females, 0 for males).
2. Label Encoding:
   • Assign unique numerical values to each category in a column.
   • Simpler approach, but the assigned numbers don’t inherently have any meaning beyond representing categories.
   • Example: Education level (high school = 0, bachelor’s = 1, master’s = 2).

Choosing the Right Technique:

• Consider model type: Some models work better with dummy variables (e.g., linear regression), while others can handle label encoding (e.g., decision trees).
• Avoid numerical assumptions: Label encoding can introduce unintended ordinal relationships if not used carefully.
• Consider data characteristics: The number of categories and their relationships can influence the choice.

Additional Considerations:

• Ordinal Encoding: For ordinal categorical data (categories with a natural order), numerical values can be assigned based on their order.
• Target Encoding: Encoding categories based on their relationship with the target variable in predictive modeling tasks.

Best Practices:

• Experiment with different techniques: Evaluate model performance with various encoding methods to find the best fit.
• Consider domain knowledge: Incorporate understanding of the data and business problem to guide encoding choices.
• Document decisions: Clearly record the encoding methods used for reproducibility and understanding.

Remember:

• Data encoding is a crucial step in preparing categorical data for analysis and modeling.
• Choose encoding techniques thoughtfully to ensure accurate and meaningful results.

Categorical data can be grouped on its qualities, thus enabling data professionals to store and identify it based on its category.

True

Categorical data can be grouped depending on its qualities, thus enabling data professionals to store and identify it based on its category.

Have you ever tried to charge your
phone with the wrong type of plug? It is a frustrating problem but
definitely not an insurmountable one. You will probably experience a similar
frustration when you work with categorical data. Categorical data is data that is
divided into a limited number of qualitative groups. For example, demographic information
tends to be categorical, like occupation, ethnicity and
educational attainment. Another way to think about categorical
data is data that uses words or qualities rather than numbers. As a data professional, you will likely work with datasets
that contain categorical data. Categorical data entries can typically
be identified quickly because they are often represented in words and
have a limited number of possible values. Many data models and
algorithms don’t work as well with categorical data as they
do with numerical data. Assigning numerical representative
values to categorical data is often the quickest and most effective way
to learn about the distribution of categorical values in a data set. There are some algorithms that work well
with categorical variables in word form, like, decision trees, which you’ll
learn about in another course. However, with many data sets, the categorical data will need to
be turned into numerical data. This conversion is often essential for predicting classifying forecasting and
more. There are several ways to change
categorical data to numerical. In this video we will focus
on two common methods, creating dummy variables and
label encoding. One way to work with categorical
data is to create dummy variables to represent those categories? Dummy variables are variables
with values of 0 or 1, which indicate the presence or
absence of something. It helps to open a data set
with dummy variables already created to understand
what their function is. You’ll find in this data set that for any value that has been determined
to be mild, a 1 has been input. All other values in the mild
column are labeled with 0s. The same goes for
other categories and values. You can think of it as the one represent
a yes, and the zeroes representing no. Dummy variables are especially
useful in statistical models and machine learning algorithms. Another way to work with categorical
data is called label encoding. Label encoding is a data transformation
technique where each category is assigned a unique number
instead of a qualitative value. For instance, let’s imagine you had a data
set about mushrooms, in which you wanted to understand the distribution
among different types of mushrooms. Imagine you’ve been given a data set about
mushrooms with a column titled type, and the options of black truffle,
cremini, king trumpet, button, hedgehog, morel,
portobello, toadstool or shiitake. You could use label encoding to transform
each mushroom type into a number. Black truffle would be changed to a 0, button would become a 1,
cremini a 2, hedgehog, a 3, king trumpet a 4,
morel 5, Portobello 6, shiitake 7, and
lastly toadstool would be 8. So why do data professionals
use label encoding? Data is much simpler to clean, join and group when it is all numbers,
it also takes up less storage space. An algorithm or model typically runs
smoother when we take the time to transform our categorical
data into numerical data. For example, let’s say you try to run
a prediction model using the mushroom dataset which will try to anticipate
the percent chance a new mushroom introduced to the data is a Portobello. If we try to create a model without
first performing the label encoding, the prediction model will not function. Of course there will be models and algorithms in which you may not
want to perform label encoding. Let the business need be
your guide on whether or not you’ll need to perform label encoding. The type of model or algorithm you
choose will determine whether or not you encode labels. You’ll learn more about algorithms and
models in an upcoming course. Much like making sure you have the right
phone charger to plug into your phone, as a data professional, you need to make sure your data is
ready to use with models or algorithms.

Here’s a tutorial on label encoding in Python for categorical data:

What is Label Encoding?

• It’s a technique for transforming categorical data (text labels) into numerical data (integers).
• This is often necessary for machine learning algorithms that require numerical inputs.

Why Use Label Encoding?

• Machine Learning Compatibility: Many algorithms can’t directly handle text-based categorical data.
• Visualization: Numerical representations can be easier to visualize and plot.
• Feature Engineering: Label encoding can be a step in preparing categorical features for modeling.

Steps for Label Encoding in Python:

1. Import Libraries:

import pandas as pd
from sklearn.preprocessing import LabelEncoder

1. Load Data:

df = pd.read_csv("your_data.csv")

1. Identify Categorical Columns:

categorical_cols = ["color", "size", "category"]  # Replace with your column names

1. Create LabelEncoder Instances:

encoders = {}
for col in categorical_cols:
    encoder = LabelEncoder()
    df[col] = encoder.fit_transform(df[col])
    encoders[col] = encoder  # Store encoders for later use

1. Access Numerical Representations:

print(df["color"])  # Will now show integers instead of text labels

1. Reverse Transformation (Optional):

original_labels = encoders["color"].inverse_transform([0, 1, 2])  # Example

Key Points:

• Label encoding assigns arbitrary integer values to categories.
• It doesn’t imply any inherent order or relationship between categories.
• Consider ordinal encoding or one-hot encoding for cases with inherent order or multiple categories per feature.
• Handle missing values appropriately before encoding.

Example:

Python

data = ["red", "green", "blue", "red", "green"]
encoder = LabelEncoder()
encoded_data = encoder.fit_transform(data)
print(encoded_data)  # Output: [0 1 2 0 1]

Fill in the blank: A heat map uses _____ to depict the magnitude of an instance or set of values.

colors

A heat map uses colors to depict the magnitude of an instance or set of values. Heat maps are a type of data visualization that is useful for showing the concentration of values between different data points.

In a previous video, you learned about
the importance of label encoding, also known as transforming
categorical data into numerical data. Now it’s time to open our python
notebooks and learn how to do it. Let’s continue with the same NOAA
dataset we’ve been using for lightning strike counts. For this notebook, we’ll look at
data from 2016, 2017, and 2018. Let’s start by importing
python libraries and packages. For this notebook,
let’s import date time pandas, seaborn and pi plot from matt plot lib. As you learned previously,
start by converting the date column into date time,
which makes it easier to manipulate. Next, create a column called month. Like we did earlier, we will use str.slice to cut the month names down to
only the first three letters. When working with this data, it is helpful to make sure the month
names stay in chronological order. Our next line of code then is months, which simply defines
the group of months in order. You’ll find this helpful on the next line
of code where we use the pandas function categorical to group the number of strikes
column by the month column because we have three years worth of data. Let’s also create a column
listing the year. To do this, we name the column year,
using the date time function strf time. This function will return a string
representing only the year in the original date column. Inside the argument field,
we input percent sign y indicating we only want the year from
the date time string. Finally, let’s create a data
frame called df by month, which groups the number of strikes
first by year and then month. The last part of the code here is
making sure that the number of strikes is added together for
each month using the sum method. We’ll tack on the reset index function
to tell the notebook to recount the rows from zero without
using its initial order. When we look at the top of the data frame,
using the head function, we get the first five rows of
data with columns of year, month, and number of strikes. Let’s create a column for
categorical variables called strike level. In this column, we will group or
bucket the total number of strikes into categories Mild,
scattered, Heavy, and Severe. This is how we perform
label encoding in python. To do this will create a new column in R df by month data
frame called strike level. This new column will be coded
using the pandas function qcut the queue in this function
refers to quantiles. Meaning that when using
these pandas function, you can cut the data into
four equal quantized. First parameters, we first input
the desired column we want to cut into quantiles, which will be
the column called number of strikes. Then we input the number of quantiles we
want, which is 41 for each classification. We type in those classifications
under the label parameter, Mild, scattered Heavy, and Severe. Let’s use the head function to
find out how our code modified the data frame df by month. It may seem counterintuitive to
create categorical data in order to then transform it to numerical data. But this process will allow us to
segment the data into useful chunks, which can then be plotted on
a graph in interesting ways. Not to mention, string categories
are easier to understand in a data visualization than numbers. One other column which will help us
later is to assign numerical values to the strike levels. We just defined as mild,
scattered, heavy, and severe. We’ll call this new
column strike level code. We’ll define it by taking the column
strike level and adding .cat.codes. Cat codes is a pen dysfunction
that takes categories and assigns them a numeric code. In this case, when we run the cell we
find that mild has been assigned a 0, scattered assigned a 1,
heavy a 2, and severe a 3. And that’s how we perform label encoding. Having both the categorical terms for
different levels of lightning strikes and the numeric codes to go with them,
we can create data visualizations or models that are straightforward
to interpret. One other helpful way to work with
categorical data is to create dummy variables to
represent those categories. As you’ll recall from a previous video,
dummy variables are variables with values of 0 or 1, which indicate
the presence or absence of something. The pandas function will use
to achieve this in our python notebooks is called get dummies. To clarify, new columns of ones and
zeros are also known as dummies. The get_dummies function converts
these categorical variables, mild, scattered, heavy, and
severe into numerical or dummy variables from the column
called strike level. If we run the cell, you’ll find this
function creates four new columns and puts a one in any row where
the number of lightning strikes falls into the labeled category,
and a zero for anything else. Now that we have our new columns of
categorical data, numeric data, and dummy variables, let’s discover what you can
potentially learn from these groupings. To do that, let’s plot our data. We’ll first create a data
frame called df_by_month_plot. Let’s reshape the data by using
the pandas function, pivot. Pivot allows us to reshape our
data frame into any given index or set of values we want based on
the parameters inside the pivot function. Those parameters are index columns and
values in that order. To help us with the visualization,
let’s use year in the index field, month in the columns field, and
strike level code in the values field. If we use our head function,
we find all three years in the rows, the months in the columns and the strike level categories in each month,
just like we coded. Next, let’s visualize our data
using what is called a heatmap. So we have a better idea of where most
of the lightning strike values are. A heatmap is a type of data visualization
that depicts the magnitude of an instance or
set of values based on two colors. It is a very valuable chart for showing the concentration of values
between two different points. Let’s create a heatmap using python. Using the heatmap plot from seaborn, we can place df_by_month_plot
in the parameter field for the data along with sea map equals blues. This sea map definition will give
us a preset color gradient from dark blue to white. Next, we fill in some of the parameters to
help with understanding the visualization. In this case, we’ll use the standard
collection zero from seaborn for our color bar. This will just give us
a preset gradient color bar, which saves us a lot of time from having
to code our own colors on a color bar. Next, we’ll set the color bar
ticks to zero through three and label those ticks according to
our four strike levels, severe, heavy, scattered, and mild. Lastly, when we code the show function,
we are given a beautiful heatmap. Using the heatmap, we can find the most severe lightning
strike months across all three years. Thanks to Pandas and Seaborn
transforming data into categories and back into numerical data is possible with
just a few lines of code as we discussed. It’s easier to plot a visualization
by using strike level code as the base values. The categories we created of mild,
scattered, heavy, and severe became
the labels on our heatmap plot. Label encoding can be as simple as that,
and it is an essential skill for data professionals.

Input Validation and Joining in EDA: A Practical Tutorial

Data is the lifeblood of analysis, but dirty data can lead to messy conclusions. Input validation and joining are two crucial practices in Exploratory Data Analysis (EDA) that ensure your data is reliable and ready for the big questions. This tutorial will equip you with the tools and techniques to validate your data and combine it effectively for deeper insights.

Part 1: Input Validation – Ensuring Data Integrity

1. Know your Data: Before diving in, understand the data source, format, and expected values. Ask questions like:
   • What types of data are present (numerical, textual, dates)?
   • Are there any expected ranges or limitations for these values?
   • How is the data structured (columns, rows, format)?
2. Inspect with Care: Use visual and statistical tools to identify anomalies:
   • Descriptive statistics: Calculate measures like mean, median, standard deviation to spot outliers.
   • Distribution plots: Visualize the spread of data using histograms and boxplots.
   • Data profiling: Look for missing values, duplicate entries, and inconsistencies in format.
3. Cleanse and Correct: Address the issues you identified:
   • Impute missing values: Use appropriate methods like mean or median imputation.
   • Standardize formats: Ensure consistent date formats, units of measurement, etc.
   • Handle outliers: Decide whether to remove, adjust, or investigate further.
4. Repeat and Refine: Validation is an iterative process. Re-run checks after cleaning and throughout your analysis to maintain data integrity.

Part 2: Joining – Combining Data for Richer Insights

1. Identify the Need: Determine what questions you want to answer and which datasets hold relevant information.
2. Find the Common Thread: Identify a unique key (e.g., customer ID, product code) present in both datasets to link them.
3. Choose the Join Method: Depending on your needs, use different join types:
   • Inner Join: Keeps matching rows from both datasets.
   • Left Join: Keeps all rows from the left dataset and matching rows from the right.
   • Right Join: Keeps all rows from the right dataset and matching rows from the left.
4. Verify the Outcome: Check for unexpected duplicates, missing information, and alignment of joined data points.

Bonus Tips:

• Automate with scripts: Develop Python or other language scripts to streamline repetitive validation and joining tasks.
• Document your process: Clearly explain your data cleaning and joining decisions for future reference and reproducibility.
• Collaborate with others: Peer review your validation and join methods to benefit from different perspectives.

Practice Makes Perfect:

This tutorial provides a framework, but the best way to master these skills is to practice. Load datasets from real-world sources, apply the techniques we discussed, and see how clean and joined data empowers your analysis. Remember, validated and joined data is the foundation for accurate insights and impactful conclusions!

Additional Resources:

• Pandas DataFrame documentation: https://pandas.pydata.org/docs/
• Python libraries for data validation: https://pypi.org/project/validation/
• Tutorials on data joining in Python: https://www.w3schools.com/python/pandas/ref_df_merge.asp

When I think about input validation and
the EDA practice of joining, I like to think about vegetables. Allow me to explain, when you’re at the market picking out
leafy greens and root vegetables, you check that they’re fresh first,
right, and not just at the store. You also check on them at home
when they’re in the refrigerator. Plus you probably ensure that
they’re still edible one more time before you cook and eat them. Lastly, if for some reason
the vegetables have gone bad or you didn’t buy enough for
a recipe you would need to add more. Performing EDA on a data set is similar to
checking for the freshness of vegetables. You are searching, exploring and checking that the data is
as error free as possible. You should check and recheck your data
sets to make sure that they are correct. As I mentioned before as
a data professional you should know your dataset thoroughly. One way to make sure you know
a data set is validating it, which is as you recall,
one of the six practices of EDA. There are many different ways
to validate a set of data, but here we’ll discuss Input validation. Input validation is the practice
of thoroughly analyzing and double checking to make sure data is
complete, error free, and high quality. Input validation is intended
to be an iterative practice, meaning you should perform it again and
again in between and during the other five practices of EDA,
which are discovering, structuring, cleaning,
joining and presenting. Most often as a data professional you
will perform input validation when starting a new analysis project or
getting familiar with a new data source. We will discuss more about the how of
performing input validation later in the program. Now we will focus on the why and the what? Why should we take
the time to validate data? What exactly are we looking for? When we validate data, we help make
more accurate business decisions and we improve complex model performance. Think about it like this if a gourmet chef
doesn’t double check the vegetables for freshness before cooking a dish
the food could taste horrible or worse, make people sick. It is much the same with data. The more careful we are in checking and
rechecking our data after each manipulation of the data during EDA, the
less likely we are to have problems later. Clean and validated data can help
prevent future system crashes, coding issues or wrong predictions. So, while we are performing our validation
work, what is it that we’re searching for? No two data sets are alike. There will be different things
to check for based on the type. Here’s some questions to
consider when validating data. Are all the data entries in
the same format, for example, are people’s ages expressed as
solitary numbers like 23 and 47 or within a range like 18 to 35 and
35 to 50, are all entries in the same range? For instance, in finance are some of the
values expressed in thousands of euros and others in millions of euros. Finally are the applicable data entries
expressed in the same data type. For example, are all the dated
entries expressed in the same format of month, day or year? While asking these questions or
while performing media in general, you may find that the data you have
been given is not sufficient to answer the business questions that
you have been tasked with. For example, let’s go back to the
comparison we made earlier in the video to a gourmet chef and his vegetables. Imagine that the gourmet chef is
introducing a new recipe to their restaurant. They’re not sure how many vegetables
will be needed for the new dish. In the end the chef realizes halfway
through that day that they did not account for additional vegetables to
be used in vegan and vegetarian dishes. It is at that point the chef
would buy more vegetables. The EDA practice of joining
is much the same principle. As you’ve learned, joining is different
from the structuring technique merging. While asking these questions or
while performing media in general. You may find that the data you have
been given is not sufficient to answer the business question. Joining is the process of augmenting data
by adding values from other data sets. The practice of joining will be most
useful if we validate the data to ensure formatting and data entries
aligned and are the same data type. For example, while adding the new
vegetable, the chef will want to make sure the consistency and taste are similar
to the vegetables that they’ve been using all day, otherwise
the dish may not turn out the same. You will need to use your own logic,
common sense and experiences to understand the ways
in which you should join or validate each particular
data set you work on. There won’t be a rigid process to show
you down to the detail how to handle each data file. Data science takes a lot
of analytical thinking and shifting of perspectives to be
thorough in your EDA and validation. Experience and effort will be the best way to improve
your performance in analytical thinking. We’ll also be exploring examples
in our Python notebooks as well. Keeping your validation practices in line
with the PACE workflow will also help you keep a focus on ethics. Validation should be about cleaning and
correcting data for the sake of quality and correctness. You probably won’t be surprised to learn
that the EDA practice of validating fits squarely in the analyze phase
of our PACE workflow, but is also a practice you should use for
all four phases. Remember, PACE stands for plan,
analyze, construct and execute. This means that what and how we join and validate data should be in alignment
with the plan phase of the workflow. For example, if your task is to discover
which week within a given month is most profitable for a business,
it will be important that you check and recheck that you’ve grouped the revenue
dates into weeks correctly. Your analysis won’t be effective, if you think you’ve grouped revenue by
week but actually done it by day or month. When you look to validate your
data to this level of detail, it will help you towards
meeting your PACE goals, specifically for
the construct and execute phases. Before we end, there’s one more thing
that can help you with validating and joining aspects of EDA. Look to your peers and
managers to help you. A peer reviewed data set is one of
the best ways to ensure that you check yourself on bias,
keep ethics as a focus and ensure you are on track
with the PACE workflow. Remember, validating data is like checking
your vegetables to find if they’re the right choice for the recipe. It may take extra time up front but
it may save you from headaches and stomachaches down the road.

Here’s a tutorial on Input Validation in Python with NOAA Data:

Introduction

• Define input validation as the process of ensuring data quality and completeness.
• Highlight its importance for accurate analysis and publication.

Setting Up

1. Import necessary libraries:

import pandas as pd
import matplotlib.pyplot as plt
import plotly.express as px
import seaborn as sns

2. Load the NOAA lightning strike dataset:

df = pd.read_csv("noaa_lightning_data.csv")

Data Analysis

1. Check data types:

df.dtypes

• Convert the “date” column to datetime if needed:

df["date"] = pd.to_datetime(df["date"])

2. Identify missing values:

df.isnull().sum()

3. Review variable ranges and distributions:

df.describe(include="all")

Date Validation

1. Check for missing calendar days:

full_date_range = pd.date_range(start="2018-01-01", end="2018-12-31")
missing_dates = full_date_range.difference(df["date"])
print(missing_dates)

• Document any missing dates.

Data Visualization

1. Create a box plot of the number of strikes:

sns.boxplot(data=df["number_of_strikes"])
plt.show()

• Remove outliers for better interpretation:

sns.boxplot(data=df["number_of_strikes"], showfliers=False)
plt.show()

Location Validation

1. Create a DataFrame with unique latitude and longitude points:

df_points = df[["latitude", "longitude"]].drop_duplicates()

1. Plot the points on a map using Plotly Express:

fig = px.scatter_geo(df_points, lat="latitude", lon="longitude")
fig.show()

Conclusion

• Emphasize the importance of input validation for reliable analysis.
• Encourage practicing these techniques for data quality assurance.

In this video, you’ll perform
input validation in Python. Remember, input validation is the practice of
thoroughly analyzing and double-checking to
make sure data is complete, error-free
and high-quality. Just like checking your
vegetables before you cook them. With that knowledge
fresh in your mind, it’s time to discuss how to perform input
validation in Python. We’ll focus our validation
on the NOAA data The goal will be to
check for errors and prepare this dataset
for publication. Let’s open our Python
notebooks and get started. First step, as usual, we will import the
Python libraries and packages that we need
for input validation. You’ll notice we have
pyplot from matplotlib, pandas, plotly.express,
and seaborn. Next, with our lightning
strike data libraries and packages all imported
in our notebook, let’s use the head function
to take a look at what we have in terms of columns
and the top five rows. You’ll find our familiar
lightening strike dataset also includes a fourth
and fifth column, longitude and latitude
respectively. If we were to check the
datatypes now using df.dtypes, we would discover the date
column is a string datatype. As we did previously, let’s convert the
column to datetime. One thing we should include in our input validation is a
check for missing data. You’ll recall from earlier that we likely will have already done this check for missing entries during the cleaning part of EDA. The definition of
validate assumes an action or process has
already been completed, so if we input isnull.sum
into our notebook in order to give us missing
values for the second time. It is a practical and not
redundant thing to do. After running the code, we find that there are
indeed no missing values. Next, for this dataset, let’s review the ranges
of all the variables. In this review,
we will check the highest and the lowest
values of each column, and the overall
distribution of values. To do this, we’ll use
the describe function, which you may recall
from previous video. In the argument
field will use the parameter includes equal, all. This will tell the
notebook to include every parameter covered
by the describe function. You’ll notice NaN
values in some of the describe fields like
unique, last and max. Remember that these are
not fields in the dataset, but rather a summary
table of attributes. For example, you’ll
notice that there are NaNs in the date
column for mean, min, 50 percent, and max. These are to be expected
because dates are not datatypes that
would be averaged. Speaking of dates, let’s
double-check our date column. First, are there any
calendar days missing? We know from prior EDA
work on this dataset that the date column should include
every day of each year. Let’s confirm each
day is listed by designing a calendar index, we’ll call it full date range. We’ll then use the pandas date range function with the
first and last dates of 2018 in the argument field under the start and
end parameters. After that, let’s add
full_date_range.difference. Then in the argument
field we’ll add df, and in the brackets,
the column header date, which will limit
this function to checking only the date column. The result is
interesting. There are four consecutive days
missing in June of 2018, then two consecutive
days in September, and two consecutive
days in December. This finding would
be something to investigate or to question
the owner of the data, to find out the reason why. Given that the number of missing days is
relatively small, you can complete your
analysis by making a note of the missing
days in the presentation. This will ensure that anyone who analyzes your visualization or presentation will
know that the data depicted doesn’t include
those missing dates. Now that we’ve done some
initial validation, let’s get a better
sense of the range of the total numbers
of lightening strikes. If we use a simple box plot
data visualization with number of strikes as our data parameter in
the argument field, we’ll find that the
distribution is a very skewed. Most days of the year have less than five
lightning strikes, but some days have
more than 2,000. As you recall from
a previous video, we can remove the outliers from the boxplot visualization to help us understand where the majority of our
data is distributed. We can do that by
adding show flyers equals false into
the argument field. The result is much
easier to interpret. From what we know of
lightning strikes in the North America regions
of the United States, Mexico, and the Caribbean, this distribution
is conceivable. If the highest distribution of strikes were all in the 2000s, we might be a little
more skeptical. The last bit of input
validation we will do is to verify that all of the latitudes and
longitudes included in this dataset are in
the United States. This will ensure we don’t have any mistakes in our
geographic values. To do that, we’ll first create a new DataFrame that removes
all the duplicate values. We’ll type df_points equals df, then latitude and longitude
in double brackets. Following that, we’ll
add.drop_duplicates. The reason we want to drop duplicate points for the
latitude and longitude is that we don’t need to check a particular location
on a map twice. After running that code, we have a new
DataFrame that only includes two columns,
latitude and longitude. Within those two columns, we find that there are no
duplicate data points. Finally, we will plot
these points on a map. Here’s a tip. When
you’re working with hundreds of thousands of data
points plotted on a map, it takes a lot of
computing power. To make it a shorter runtime, we’ll use the Plotly
express package. The package is designed to keep runtimes as low as possible. To plot these latitude and
longitude points on a map, we use the scatter geo
plot from Plotly Express. Within the argument field, we’ll fill in the
proper parameters for a scatter geo plot. In this case, we’ll first input the DataFrame name df points,
followed by first lat for latitude then lon for longitude. The last code is to
show the scatter plot. After running this cell, you’ll notice the
runtime is still slow because of the sheer
volume of the data. Most data professionals use input validation regularly, so these are important concepts to learn and important technical skills to practice.

As you learn data sets can be busy and
chaotic. There are many different
things happening in them. It can be difficult to figure
out where to focus and how to uncover those stories
buried beneath the surface? In this section of the course, you learn
some important strategies to help you correct mistakes and
discover hidden data stories. We focused our story finding efforts by
learning the EDA practices of cleaning, joining and validating. We discussed a lot of ways to
engage in those practices. We focused on working with
missing values and outliers, transforming categorical into
numerical data and input validation. And we reminded ourselves not to
forget to visualize our data in python to help further our understanding. We discussed identifying missing data and
outliers in python and why it’s important both from an ethical perspective
and a business sense to find them. We considered the difference between
categorical and numerical data and why transformation using
python is important. As for input validation,
you learned what it is? Why it’s important and
how to perform it using python? We also discussed some workplace skills
along the way such as understanding when to communicate with stakeholders or other subject matter experts
about missing values. We also reviewed ethical considerations
you should make when performing cleaning, joining and validating work. You learn why these practices
are vital when performing EDA an a data set as part
of the pace workflow. EDA can be an exciting process,
thorough EDA not only saves time and energy on future processes but
it also helps you find the trends, patterns and stories in the data. Later, you’ll learn how to use tableau to
design and present data visualizations to key stakeholders and
business managers in a business setting. In the meantime, you’ve learned some amazing skills that
data professionals use almost every day. The ability to perform careful EDA on a
dataset in python is an essential building block in a data professionals career. Great job on your work so far.