- Assess whether the information can be used to meet the needs of the organization.
- Assess the data quality
- Check the potential for using the data in a new or different application.
- Assess the risks that any kind of data processing, storage, or management might have on the data.
- Have an overall view of managing and governing data for the warehouse.
If you stay in the ETL business, you will find that data profiling often becomes the backbone of your work in planning, designing, implementing, and managing a data warehouse. In order to guide you in your projects, you can follow these top four data profiling best practices, all very helpful tips in this line of work. They can be applied to your circumstances, too.
Data Profiling Best Practice 1: Do it before data warehousing
Do data profiling before data warehousing. This is probably the first and foremost data profiling best practice that you should know. When you find that your company or organization needs to set up a data warehouse, then your first order of business is to profile the information that you now have, before embarking on any of the extract-transform-load processes. This first step allows you to look long and hard objectively into the information that you have and to see what its quality is. This first data profiling step provides metadata that you will usually need when designing the rest of your data warehouse. This data profiling best practice will ensure that your ETL best practices are as efficient as they can be. In most cases, a data profiling done before the actual data warehousing project, will even tell you if the project is worth pursuing—saving you thousands of dollars in manpower hours.
Data Profiling Best Practice 2: Know the full array of tools
It is necessary for you to know the full array of tools that you can use to create your data profiling. Best practice is to get an idea of what programs or platforms are in the market then compare their functionality, price points, and usage so that you will have an idea of which tool is best suited for your data profiling purposes. Some tools available in the market are DataFlux, Trillium, and Informatica and even the basic data profiling tools that come with SQL Server. Currently, some providers have also released automated profiling tools. Most ETL experts agree that the last option is already suitable for basic, light data profiling work. For the more advanced kinds of projects, however, a tool dedicated or mainly designed for data profiling work is best. Aside from the kind of work, settling on a data profiling tool should, of course, be based on the level of standardization that you want. What you should do as a data profiling best practice is to get on to the Internet and faithfully do your research about each of these tools and read user/programmer feedback. Remember that choosing a data profiling tool really is a commitment to a kind of standard and style for data profiling.
Data Profiling Best Practice 3: Combine data profiling with cross-checking
If you really want to get the most out of your data profiles, you need to remember that the information you’ll be working with does not only come from inside, but must also be validated by outside sources. That means that you have to cross-check the facts that you have from data profiling with the feedback and other information that you are getting from other sources, which may not be so easy to tabulate and input in a data warehouse. These outside ways of checking data profiles may come from customer feedback, simple questionnaires for your stakeholders, trends relating or corresponding to the items in your data profile, or maybe a general survey of what your industry benchmarks are. This data profiling best practice can help you check the results of your inside work, as well as possibly point out errors or open up new ways of looking at your available data—all for the better understanding of information about your company.
Data Profiling Best Practice 4: Review segmentation
As a last data profiling best practice, you must continue running this process even after you have designed your data warehouse. Profiling data regularly can help you look for new ways of interpreting the data that you have, which usually brings a fresh insight into how the company or organization is doing, revealing changing trends and sounding off on any need to innovate or adapt so that the company will perform better. More than just regular, periodic data profiling, however, you also have to regularly review the way you segment the information for your data profiling work. This means that you have to cut through your data in new ways so that you get fresh insights into the same information. This will help you broaden or focus your data segmentation so that the metadata will give you either a widened or a specified look into your market and your business transactions. Sometimes, adjusting your data profile segmentation can mean just doing the data profile on a relevant part of information within your data warehouse. This is done so that you can be more efficient at studying the different related information or information chains. Data profiling best practices will make the data identification process faster and will yield more information about the more important aspects of your matrix of data.