“The truth is not for all men but only for those who seek it.”
― Ayn Rand
Contents
1. Introduction
The Series is one of the most common pandas data structures. It is similar to a python list and is used to represent a column of data. After looking into the basics of creating and initializing a pandas Series object, we now delve into some common usage patterns and methods.
2. Series Information
After a Series is created, it is most important to look into various details of its structure. These include the size of the series, whether there are NaNs in it, etc. Here are some commonly used methods which help clarify the situation.
2.1. Size of the Series
There are several methods to determine how big the Series is.
The first is the attribute shape which returns a tuple.
a = pd.Series(random.sample(xrange(100), 6)) print a.shape # prints (6,)
We also have the count() method which returns the size of the Series as an integer.
print a.count() # prints 6
However, note that count() only reports the number of non-NaN elements, while shape reports both.
Another attribute for getting the count of elements is size. It reports the count as an integer, and includes NaN elements if any.
a = pd.Series(random.sample(xrange(100), 6)) print 'count of a =>', a.count(), '\n' b = a.append(pd.Series(np.nan, index=list('abcd')), ignore_index=True) print 'b => ', b, '\n' print 'count of b =>', b.count(), '\n' print 'shape of b =>', b.shape, '\n' print 'size of b =>', b.size # prints count of a => 6 b => 0 76.0 1 92.0 2 75.0 3 60.0 4 42.0 5 44.0 6 NaN 7 NaN 8 NaN 9 NaN dtype: float64 count of b => 6 shape of b => (10,) size of b => 10
2.2. Series Details
Get some detailed stats on the Series using describe(). This method returns a Series object with the index (or labels) as shown.
x = pd.Series(random.sample(xrange(100), 6)) x.describe() # prints count 6.000000 mean 60.500000 std 30.742479 min 20.000000 25% 44.000000 50% 53.500000 75% 86.250000 max 98.000000 dtype: float64
2.3. Head and Tail
Show the first 5 or last 5 rows of the Series using head() or tail().
x = pd.Series(random.sample(xrange(100), 10)) print x, '\n' print x.head(), '\n' print x.tail(), '\n' # prints 0 24 1 39 2 56 3 77 4 81 5 26 6 8 7 87 8 34 9 68 dtype: int64 0 24 1 39 2 56 3 77 4 81 dtype: int64 5 26 6 8 7 87 8 34 9 68 dtype: int64
3. Add Elements to Series
Adding elements to a Series is accomplished by using append()
. The argument must be a single Series object, or a list (or tuple) of Series objects.
x = pd.Series(random.sample(xrange(100), 6)) print x, '\n' print 'appended =>\n', x.append([pd.Series(2), pd.Series([3, 4, 5])]) # prints 0 62 1 29 2 20 3 69 4 53 5 22 dtype: int64 appended => 0 62 1 29 2 20 3 69 4 53 5 22 0 2 0 3 1 4 2 5 dtype: int64
You might notice the odd-ball labels after appending. Each Series is appended with a default index starting from 0, regardless of whether this creates duplicate labels. One way to fix this is to specify ignore_index=True
to ensure re-labeling.
print 'appended =>\n', x.append([pd.Series(2), pd.Series([3, 4, 5])], ignore_index=True) # prints appended => 0 62 1 29 2 20 3 69 4 53 5 22 6 2 7 3 8 4 9 5 dtype: int64
What if you don’t want to re-label but ensure that append() succeeds only if the labels are unique? Keep your precious labels intact and unique by specifying verify_integrity=True.
print 'appended =>\n', x.append([pd.Series(2), pd.Series([3, 4, 5])], verify_integrity=True) # throws exception ValueError: Indexes have overlapping values: [0, 1, 2]
4. Delete Elements
You can delete elements from a Series using the following methods.
4.1. By Label
Use drop() and specify a single label or a list of labels to drop.
x = pd.Series(random.sample(xrange(100), 6), index=list('ABCDEF')) print x, '\n' print 'drop one =>\n', x.drop('C'), '\n' print 'drop many =>\n', x.drop(['C', 'D']) # prints A 67 B 18 C 1 D 54 E 38 F 3 dtype: int64 drop one => A 67 B 18 D 54 E 38 F 3 dtype: int64 drop many => A 67 B 18 E 38 F 3 dtype: int64
4.2. Duplicate Elements
Get rid of duplicate elements by invoking drop_duplicates().
x = pd.Series([1, 2, 2, 4, 5, 7, 3, 4]) print x, '\n' print 'drop duplicates =>\n', x.drop_duplicates(), '\n' # prints 0 1 1 2 2 2 3 4 4 5 5 7 6 3 7 4 dtype: int64 drop duplicates => 0 1 1 2 3 4 4 5 5 7 6 3 dtype: int64
By default, the method retains the first repeated value. Get rid of all duplicates (including the first) by specifying keep=False
.
drop all duplicates => 0 1 4 5 5 7 6 3 dtype: int64
4.3. NaN Elements
Use the dropna() to drop elements without a value (NaN).
x = pd.Series([1, 2, 3, 4, np.nan, 5, 6]) print x, '\n' print 'drop na =>\n', x.dropna() # prints 0 1.0 1 2.0 2 3.0 3 4.0 4 NaN 5 5.0 6 6.0 dtype: float64 drop na => 0 1.0 1 2.0 2 3.0 3 4.0 5 5.0 6 6.0 dtype: float64
5. Replace NaN Elements
When you want to replace NaN elements in a Series, use fillna().
x = pd.Series([1, 2, 3, 4, np.nan, 5, 6]) print x, '\n' print 'fillna w/0 =>\n', x.fillna(0) # prints 0 1.0 1 2.0 2 3.0 3 4.0 4 NaN 5 5.0 6 6.0 dtype: float64 fillna w/0 => 0 1.0 1 2.0 2 3.0 3 4.0 4 0.0 5 5.0 6 6.0 dtype: float64
6. Select Elements
Select elements from a Series based on various conditions as follows.
6.1. In a Range
Use the between() method which returns a Series of boolean values indicating whether the element lies within the range.
a = pd.Series(random.sample(xrange(100), 10)) print a print a.between(30, 50) # prints 0 85 1 42 2 63 3 69 4 81 5 45 6 50 7 72 8 66 9 34 dtype: int64 0 False 1 True 2 False 3 False 4 False 5 True 6 True 7 False 8 False 9 True dtype: bool
You can use this the returned boolean Series as a predicate into the original Series.
print a[a.between(30, 50)] # prints 1 42 5 45 6 50 9 34 dtype: int64
6.2. Using a Function
Select elements using a predicate function as the argument to select().
x = pd.Series(random.sample(xrange(100), 6)) print x, '\n' print 'select func =>\n', x.select(lambda a: x.iloc[a] > 20) # prints 0 83 1 96 2 29 3 15 4 28 5 12 dtype: int64 select func => 0 83 1 96 2 29 4 28 dtype: int64
6.3. By List of Labels
Use filter(items=[..]) with the labels to be selected in a list.
x = pd.Series([1, 2, 3, 4, np.nan, 5, 6]) print x, '\n' print 'filtered =>\n', x.filter(items=[1, 2, 6]) # prints 0 1.0 1 2.0 2 3.0 3 4.0 4 NaN 5 5.0 6 6.0 dtype: float64 filtered => 1 2.0 2 3.0 6 6.0 dtype: float64
6.4. Regex Match on Label
Select labels to filter using a regular expression match with filter(regex=’..’).
x = pd.Series({'apple': 1.99, 'orange': 2.49, 'banana': 0.99, 'grapes': 1.49, 'melon': 3.99}) print x, '\n' print 'regex filter =>\n', x.filter(regex='a$') # prints apple 1.99 banana 0.99 grapes 1.49 melon 3.99 orange 2.49 dtype: float64 regex filter => banana 0.99 dtype: float64
6.5. Sub-string Match on Label
Use the filter(like=’..’) version to perform a sub-string match on the labels to be selected.
print 'like filter =>\n', x.filter(like='an') # prints like filter => banana 0.99 orange 2.49 dtype: float64
7. Sorting
Ah! Sorting. The all important functionality when playing with data.
Here is how you can sort a Series – by labels or by value.
7.1. By Index (or Labels)
Use sort_index().
x = pd.Series(random.sample(xrange(100), 6), index=random.sample(map(chr, xrange(ord('a'), ord('z'))), 6)) print x, '\n' print 'sort by index: =>\n', x.sort_index(), '\n' # prints p 37 e 44 b 93 l 75 n 4 s 83 dtype: int64 sort by index: => b 93 e 44 l 75 n 4 p 37 s 83 dtype: int64
7.2. By Values
Use sort_values() to sort by the values.
print 'sort by value: =>\n', x.sort_values() # prints sort by value: => n 4 p 37 e 44 l 75 s 83 b 93 dtype: int64
Summary
This article covered some commonly used methods to deal with a Series object. These included methods to retrieve general information about a Series, modifying a Series, selection and sorting.