Five Different Methods for Loading Data in Python - Huawei Developers

Data is the foundation of data scientists, so it is important to understand the many ways in which data is loaded for analysis. Here, we will introduce five Python data entry techniques and provide code examples for your reference.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
As a beginner, you probably know only one way to read data using the p andas.read_csv function (usually in CSV format). It is one of the most mature and powerful features, but other methods are helpful and sometimes sure to come in handy.
The way I'm going to discuss this is:
1. Manual function
2. loadtxt function
3. genfromtxtf function
4. read_csv function
5. Pickle
The data set we will use to load the data can be found here. It is called 100-Sales-Records.
Imports
We will use Numpy, Pandas, and Pickle packages, so import them.
1. Manual Function
This is the most difficult because you must design a custom function that loads data for you. You must handle the general archiving concept of Python and use it to read .csv files.
For example, let's do this on 100 sales record files.
Well, what's this? ???It seems a bit complicated code!!! Let's break it gradually so that you can understand what's going on and apply similar logic to read your own .csv file.
Here, I created a load_csv function that uses the path of the file to be read as a parameter.
I have a list named data that will have my CSV file data, and the other list col will have my column name. Now, after manually checking the csv, I know that the column names are in the first row, so in my first iteration I have to store the data in the first row in col and the rest in data.
To check for the first iteration, I used a Boolean variable called checkcol, which was false, and when it was false in the first iteration, it stored the data in the first row in col, and then set checkcol to True, so we processed the data list and stored the rest of the values in the data list.
The main logic here is that I iterate through the file using functions in readlines() Python. This function returns a list of all the rows in the file.
When reading a title, it detects a new line as a \n character, or a line-terminal character, so I use the str.replace function to remove it.
Since this is a .csv file, I have to comma according to different things, so I'll run a string each with string.split("). For the first iteration, I'll store the first row, with the list containing column names called col. Then I append all the data to a list called data.
To read the data better, I return it as a data box format because it is easier to read data boxes than numpy arrays or python lists.
Output
What are the pros and cons of doing this?
The important benefit is that you have all the flexibility and control over the file structure and can read and store it in any format and way you want.
You can also use your own logic to read files that do not have a standard structure.
Its disadvantage is that, especially for standard types of files, it is complex to write because they are easy to read. You must hard-code logic that requires trial and error.
Use it only if the file is not a standard format or if you want flexibility and read it in a way that the library does not provide.
2. Numpy.loadtxt function
This is a built-in function in Numpy, the famous digital library in Python. Loading data is a very simple function. This is useful for reading data of the same data type.
This feature is hard to read when the data is more complex, but it is powerful when the file is simple.
To get a single type of data, download the virtual dataset here. Let's jump to the code.
Here, we simply use the loadtxt function as a ',' in the input delimiter because this is a CSV file.
Now, if we print df, we'll see the data in a fairly good numpy array that we can use.
Because of the large amount of data, we printed only the first five lines.
An important aspect of using this feature is that you can quickly load data from a file into a NumPy array.
The disadvantage is that you cannot have other data types or missing rows in the data.
3. Numpy.genfromtxt ()
We'll use the data set, the 100 Sales Records.csv data set used in the first example, to prove that it can contain multiple data types.
Let's jump to the code.
To see it more clearly, we can see it in data box format, that is
What's this? Oh, it skips all columns with string data types. What do we do?
Simply add another dtype parameter and set dtype to None, which means that it has to take care of the data type of each column itself. The entire data is not converted to a single dtype.
Then output
It's much better than the first one, but the column header here is Row, and to make it a column header, we have to add another parameter, Name, and set it to True so that it will use the first row as the column header.
i.e.
1
df3 = np.genfromtxt('100 Sales Records.csv', delimiter=',' , dtype=None, names=True, encoding= 'utf-8')
We can print it as
4. Pandas.read_csv()
Pandas is a very popular database manipulation library, which is very common. Read_csv() is one of the most important and mature features that can easily read any .csv file and help us operate. Let's work on a dataset of 100 sales records.
This feature is easy to use and is therefore very popular. You can compare it to our previous code and check it.
Guess what? Done. This is actually so simple and easy to use. Pandas.read_csv certainly provides many other parameters to adjust our dataset, for example in our convertcsv.csv file, we don't have column names, so we can read them as
We can see that it has read the CSV file without the title. You can view all other parameters in the official documentation here.
5. Pickle
If your data is not in a good format that humans can understand, you can use pickle to save it in binary format. You can then easily reload it using the pickle library.
We'll get a CSV file of 100 sales records and first save it in pickle format so we can read it.
This will create a new file test.pkl containing pdDf from the Pandas header.
Now to open it with pickle, we just need to use the pickle.load function.
Here, we have successfully loaded the data from the pickle file in pandas. DataFrame format.
This article translated from https://bbs.huaweicloud.com/blogs/209773
For details about Huawei developers and HMS, visit the website.
HUAWEI Developer Forum | HUAWEI Developer
forums.developer.huawei.com

---

Interesting and can be useful.