Weird Null Checking Behaviour By Pd.notnull

This is essentially a rehashing of the content of my answer here. I came across some weird behaviour when trying to solve this question, using pd.notnull. Consider x = ('A4', nan)

Solution 1:

Here is the issue related to this behavior: https://github.com/pandas-dev/pandas/issues/20675.

In short, if argument passed to notnull is of type list, internally it is converted to np.array with np.asarray method. This bug occured, because, if no dtype specified, numpy converts np.nan to string(which is not recognized by pd.isnull as null value):

a = ['A4', np.nan]
np.asarray(a)
# array(['A4', 'nan'], dtype='<U3')

This problem was fixed in version 0.23.0, by calling np.asarray with dtype=object.