EDA involves the following steps:<\/p>\n
- \n
- Calculating mean, median, mode and quartile values of the data in each column of the data set.<\/li>\n
- Visualising the data and observing how the values are distributed.<\/li>\n
- Identifying outliers (data that do not fall within the ambit of our analysis).<\/li>\n
- The range of data in each column is calculated, which can show us the magnitude of values present as well as the minimum and maximum values.<\/li>\n<\/ul>\n
Data Preparation & EDA<\/strong><\/p>\n
Data is collected from various sources and loaded to data warehouse using Pentaho+ Data Integration. The Pentaho Plus Data Integration Tool performs the cleansing, transformation, applying rules and stores in data warehouse.<\/p>\n
This data is further used for the exploratory data analysis, creating data pipeline and building model.<\/p>\n
Uni-variate plots<\/strong> help in understanding the nature of values in each of the attributes.<\/p>\n
![\"\"](\"https:\/\/tenthplanet.in\/blogs\/wp-content\/uploads\/sites\/21\/2019\/06\/blog2-1-300x214.png\" "\\"\\"")
<\/p>\nLine plots for the variables in the data<\/strong><\/p>\n
From the above line plots, we can infer that pH and density have normal distribution, whereas SO2, sulphates and alcohol have skewed distribution. Quality shows comb distribution, which means that certain data points may have been rounded off to the nearest whole number.<\/p>\n
Bi-variate plots<\/strong> help in understanding the relationship between two plotted attributes.<\/p>\n
![\"\"](\"https:\/\/tenthplanet.in\/blogs\/wp-content\/uploads\/sites\/21\/2019\/06\/blog4-1-300x214.png\" "\\"\\"")
<\/p>\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0Quality Vs Alcohol scatter plot<\/strong><\/p>\n
From the above scatter plot, we can infer that good-quality wines usually have higher alcohol content than low-quality wines .<\/p>\n
Correlation Plot<\/strong> shows linear correlation between two variables.<\/p>\n
![\"\"](\"https:\/\/tenthplanet.in\/blogs\/wp-content\/uploads\/sites\/21\/2019\/06\/blog5-1-300x214.png\" "\\"\\"")
<\/p>\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0Correlation plot<\/strong><\/p>\n
From the above correlation plot, we can list the attributes that are strongly correlated:<\/p>\n
Quality and Alcohol (positively correlated)
\nDensity and Acidity (positively correlated)
\nDensity and Alcohol (negatively correlated)
\nFree SO2 and Total SO2 (positively correlated)
\nAcidity and pH (negatively correlated)<\/p>\nMultivariate Plots<\/strong> find relationships between more than 2 variables. They become difficult to interpret when the number of variables is greater than 3.<\/p>\n
![\"\"](\"https:\/\/tenthplanet.in\/blogs\/wp-content\/uploads\/sites\/21\/2019\/06\/blog6-1-300x214.png\" "\\"\\"")
<\/p>\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a03-D scatter plot<\/strong><\/p>\n
From the above graph, we can infer that most good-quality wines contain less amount of chlorides and an acceptable density.<\/p>\n
Conclusion<\/strong><\/h3>\n
Various types of plots and techniques are available to represent and analyse your data like Box plot<\/a>, Histogram<\/a>, Run chart<\/a>, Pareto char<\/a>t, Scatter plot<\/a>, Odds ratio<\/a> and more. It is crucial to utilise the right plot so as to grab the essence of pattern from the data.<\/p>\n
Therefore EDA is the process of obtaining any possible intuition regarding the data in the beginning itself. Skipping this process might lead to skewed data with too many outliers and missing values. This, in turn, could generate inaccurate models, lead to choosing wrong variables for the model or the wrong model itself.<\/p>\n","protected":false},"excerpt":{"rendered":"
Raw data is normally ambiguous and difficult to interpret. Cleaning it is essential in order to understand the relationships between the variables present in the data.<\/p>\n","protected":false},"author":23,"featured_media":2080,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[424],"tags":[484,468,489,478],"class_list":["post-2068","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-pentaho","tag-data-cleaning","tag-data-science","tag-data-visualization","tag-exploratory-data-analysis"],"acf":[],"_links":{"self":[{"href":"https:\/\/tenthplanet.in\/blogs\/wp-json\/wp\/v2\/posts\/2068","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tenthplanet.in\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tenthplanet.in\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tenthplanet.in\/blogs\/wp-json\/wp\/v2\/users\/23"}],"replies":[{"embeddable":true,"href":"https:\/\/tenthplanet.in\/blogs\/wp-json\/wp\/v2\/comments?post=2068"}],"version-history":[{"count":1,"href":"https:\/\/tenthplanet.in\/blogs\/wp-json\/wp\/v2\/posts\/2068\/revisions"}],"predecessor-version":[{"id":11433,"href":"https:\/\/tenthplanet.in\/blogs\/wp-json\/wp\/v2\/posts\/2068\/revisions\/11433"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tenthplanet.in\/blogs\/wp-json\/wp\/v2\/media\/2080"}],"wp:attachment":[{"href":"https:\/\/tenthplanet.in\/blogs\/wp-json\/wp\/v2\/media?parent=2068"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tenthplanet.in\/blogs\/wp-json\/wp\/v2\/categories?post=2068"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tenthplanet.in\/blogs\/wp-json\/wp\/v2\/tags?post=2068"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}