Saturday, November 26, 2022
HomeArtificial IntelligenceA Light Introduction to Serialization for Python

A Light Introduction to Serialization for Python

Final Up to date on March 2, 2022

Serialization refers back to the technique of changing a knowledge object (e.g. Python objects, Tensorflow fashions) right into a format that enables us retailer or transmit the information after which recreate the item when wanted utilizing the reverse technique of deserialization.

There are completely different codecs for serialization of knowledge, corresponding to JSON, XML, HDF5, Python’s pickle, for various functions. JSON for example returns a human-readable string type whereas Python’s pickle library can return a byte array.

On this submit, you’ll uncover easy methods to use two widespread serialization libraries in Python to serialize information objects (particularly pickle and HDF5) corresponding to dictionaries and Tensorflow fashions in Python for storage and transmission

After finishing this tutorial, you’ll know:

  • Serialization libraries in Python corresponding to pickle and h5py
  • Serializing objects corresponding to dictionaries and Tensorflow fashions in Python
  • Easy methods to use serialization for memoization to cut back perform calls

Let’s get began!

Command line arguments in your Python script. Photograph by little plant. Some rights reserved


The tutorial is split into 4 components, they’re:

  • What’s serialization and why can we serialize?
  • Utilizing Python’s pickle library
  • Utilizing HDF5 in Python
  • Comparability between completely different serialization strategies

What’s serialization and why ought to we care?

Take into consideration storing an integer, how would you retailer that in a file or transmit it? That’s straightforward! We are able to simply write the integer to a file and retailer or transmit that file.

However now, what if we take into consideration storing a Python object (e.g. a Python dictionary or a Pandas DataFrame), which has a posh construction and plenty of attributes (e.g., columns and index of the DataFrame, and the information kind of every column)? How would you retailer it as a file or transmit it to a different pc?

That is the place serialization is available in!

Serialization is the method of changing the item right into a format that may be saved or transmitted. After transmitting or storing the serialized information, we’re in a position to later reconstruct the item and acquire the very same construction/object, which makes it actually handy for us to proceed utilizing the saved object in a while as a substitute of reconstructing the item from scratch.

In Python, there are a lot of completely different codecs for serialization obtainable. One widespread instance for hash maps (Python dictionaries) which works throughout many languages is the JSON file format which is human-readable and permits us to retailer the dictionary and recreate it with the identical construction. However JSON can solely retailer fundamental constructions corresponding to record and dictionary, and it could solely maintain strings and numbers. We can’t ask JSON to recollect the information kind (e.g., numpy float32 vs float64). It additionally can’t distinguish between Python tuples and lists.

Extra highly effective serialization codecs exists. Within the following, we’ll discover two widespread serialization libraries in Python, particularly pickle and h5py.

Utilizing Python’s pickle library

The pickle module is a part of the Python normal library and implements strategies to serialize (pickling) and deserialize (unpickling) Python objects.

To get began with pickle, import it in Python

afterwards, to serialize a Python object corresponding to a dictionary and retailer the byte stream as a file, we will use pickle’s dump() methodology.

and the byte stream representing test_dict is now saved within the file “check.pickle”!

To get better the unique object, we learn the serialized byte stream from the file utilizing pickle’s load() methodology.

Warning: Solely unpickle information from sources you belief as it’s potential to for arbitrary malicious code to be executed throughout the unpickling course of.

Put them collectively, the next code lets you confirm pickle can get better the identical object:

Moreover writing the serialized object right into a pickle file, we will additionally get hold of the item serialized as a bytes-array kind in Python utilizing pickle’s dumps() perform:

Equally, we will use pickle’s hundreds methodology to transform from a bytes-array kind again to the unique object

One helpful factor about pickle is that it could serialize nearly any Python object, together with user-defined ones, corresponding to the next

The code above will print the next

Word that the print assertion within the class’ constructor will not be executed on the time pickle.hundreds() is invoked. As a result of it reconstructed the item, not recreated it.

Pickle may even serialize Python features since features are top quality objects in Python:

Subsequently, we will make use of pickle to avoid wasting our work. For instance, a educated mannequin from Keras or scikit-learn could be serialized by pickle and cargo it later as a substitute of re-train the mannequin each time we use it. The next is to indicate you ways we will construct a LeNet5 mannequin to acknowledge the MNIST handwritten digits utilizing Keras, then serialize the educated mannequin utilizing pickle. Afterwards, we will reconstruct the mannequin with out coaching it once more and it ought to produce precisely the identical consequence as the unique mannequin:

The above code will produce the output as follows, word that the analysis scores from the unique and reconstructed fashions are tied out completely on the final two traces:

Whereas pickle is a robust library, it nonetheless does have its personal limitations to what could be pickled. For instance, reside connections corresponding to database connections and opened file handles can’t be pickled. This challenge arises as a result of reconstructing these objects requires pickle to re-establish the reference to the database/file which is one thing pickle can’t do for you (as a result of it wants applicable credentials and is out of scope of what pickle is meant for).

Utilizing HDF5 in Python

Hierarchical Knowledge Format 5 (HDF5) is a binary information format. The h5py package deal is a Python library that gives an interface to the HDF5 format. From h5py docs, HDF5 “helps you to retailer enormous quantities of numerical information, and simply manipulate that information from Numpy.”

What HDF5 can do higher than different serialization format is that it shops information in a file system like hierarchy. You may retailer a number of objects or dataset in HDF5 like saving a number of recordsdata within the file system. You may as well learn a specific dataset from HDF5 like studying one file from the file system with out in regards to the different. If you happen to’re utilizing pickle for this, you will have to learn and write all the pieces every time you load or create the pickle file. Therefore HDF5 is advantageous for enormous quantity of knowledge that may’t match fully into reminiscence.

To get began with h5py, you first want to put in the h5py library, which you are able to do utilizing

Or if you’re utilizing a conda setting

We are able to then get began with creating our first dataset!

This creates a brand new dataset within the file check.hdf5 named “test_dataset”, with a form of (100, ) and a sort int32. h5py datasets comply with a Numpy syntax so you are able to do slicing, retrieval, get form, and many others. much like Numpy arrays

To retrieve at a selected index,

To get a slice from index 0 to index 10 of dataset,

If you happen to initialized the h5py file object exterior of a with assertion, keep in mind to shut the file as properly!

To learn from a beforehand created HDF5 file, you possibly can open the file in “r” for learn mode or “r+” for learn/write mode,

To prepare your HDF5 file, you should utilize teams,

One other strategy to create teams and recordsdata is by specifying the trail to the dataset you need to create and h5py will create the teams on that path as properly (in the event that they don’t exist),

The 2 snippets of code each create group1 if it has not be created beforehand, after which a dataset1 inside group1.

HDF5 in Tensorflow

To avoid wasting a mannequin in Tensorflow Keras utilizing HDF5 format, we will use the save() perform of the mannequin with a filename having extension .h5, like the next:

To load the saved HDF5 mannequin, we will additionally use the perform from Keras straight:

One purpose we don’t need to use pickle for a Keras mannequin is that we’d like a extra versatile format that doesn’t tie to a specific model of Keras. If we upgraded our Tensorflow model, the mannequin object might change and pickle might fail to provide us a working mannequin. One more reason is to maintain solely the important information for our mannequin. For instance, if we verify the HDF5 file my_model.h5 created within the above, we see these are saved:

Therefore Keras chosen solely the information which might be important to reconstruct the mannequin. A educated mannequin will include extra datasets, particularly, there are /optimizer_weights/ moreover /model_weights/. Keras will reconstruct the mannequin and restore the weights appropriately to provide us a mannequin that perform the identical.

Evaluating between completely different serialization strategies

Within the above we noticed how pickle and h5py may also help serialize our Python information.

We are able to use pickle to serialize nearly any Python object, together with user-defined ones and features. However pickle will not be language agnostic. You can’t unpickle it exterior Python. There are even 6 variations of pickle developed to date and older Python might not have the ability to eat the newer model of pickle information.

Quite the opposite, HDF5 is cross platform and works properly with different language corresponding to Java and C++. In Python, the h5py library carried out the Numpy interface to make it simpler to control the information. The info could be accessed in numerous language as a result of HDF5 format helps solely the Numpy information sorts corresponding to float and strings. We can’t retailer arbitrary objects corresponding to a Python perform into HDF5.

Additional studying

This part gives extra assets on the subject if you’re trying to go deeper.





On this submit, you found what serialization is and easy methods to use libraries in Python to serialize Python objects corresponding to dictionaries and Tensorflow Keras fashions. You’ve gotten additionally learnt the benefits and drawbacks of two Python libraries for serialization (pickle, h5py).

Particularly, you realized:

  • what’s serialization and why it’s helpful
  • easy methods to get began with pickle and h5py serialization libraries in Python
  • execs and cons of various serialization strategies



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments