Understanding feature selection and feature engineering when performing regressions#

Even with a large and complete dataset, not all of the features end up being important, and you don't always have all the fields you need. Sometimes performing calculations between two or more columns can yield a large improvement in your predictor's performance.

Read online Download notebook Interactive version

Reading in our data#

Every single person in a crash#

We'll start by reading in the list of people who were involved in an accident.

Call this dataframe people.

How often did each severity of injury show up? (e.g. not injured, non-incapacitating injury, etc)

We're only interested in fatalities, so let's create a new had_fatality column for when people received a fatal injury.

Confirm there were 1681 people with fatal injuries.

Working on Features#

Starting our analysis#

We're going to run a regression on the impact of being male vs female on crash fatalities. Prepare a dataframe called train_df with the appropriate information in it.

• Tip: What column(s) are your input, and what is your output? Aka independent and dependent variables
• Tip: You'll need to convert your input column into something numeric, I suggest using .replace
• Tip: We aren't interested in the "Unknown" sex - either filtering or np.nan + .dropna() might be useful ways to get rid of those columns

Confirm that your train_df has two columns and 815,827 rows.

Tip: If you have more rows, make sure you dropped all of the rows with Unknown sex.

Tip: If you have more columns, make sure you only have your input and output columns.

Run your regression#

See the effect of sex on whether the person's injuries are fatal or not. After we train the regression, we can use my ✨favorite technique✨ to display features and their odds ratios:

feature_names = X.columns
coefficients = clf.coef_[0]

pd.DataFrame({
    'feature': feature_names,
    'coefficient (log odds ratio)': coefficients,
    'odds ratio': np.exp(coefficients).round(4)
}).sort_values(by='odds ratio', ascending=False)

Use words to interpret this result#

Adding more features#

The actual crash data has more details - whether it was snowy/icy, whether it was a highway, etc.

Read in combined-crash-data.csv, calling it crashes, and merge it with our people dataset. I'll save you a lookup: the REPORT_NO is what matches between the two.

Examining more possible features#

How often was it wet, dry, snowy, icy, etc? What was the most common condition?

Do you feel that a Dry road condition should be the average of Wet and Snow?

The answer to that should be no, which means we can't use this data as numeric data. We want a different coefficient for each of these - I want to know the impact of dry, the impact of wet, the impact of snow, all separately.

Start by replacing each code with a proper description. I'll even include them here:

• 00 - Not Applicable
• 01 - Wet
• 02 - Dry
• 03 - Snow
• 04 - Ice
• 05 - Mud, Dirt, Gravel
• 06 - Slush
• 07 - Water (standing/moving)
• 08 - Sand
• 09 - Oil
• 88 - Other
• 99 - Unknown

But watch out, pandas read the column in as numbers so they might have come through a little differently than their codes.

Confirm you have 147,803 wet, and a few codes you can't understand, like 6.03 and 7.01.

Replace the codes you don't understand with Other.

Confirm you have 3,196 'Other'.

One-hot encoding#

We're going to use pd.get_dummies to build a variable you'll call surf_dummies. Each surface condition should be a 0 or 1 as to whether it was that condition (dry, icy, wet, etc).

Use a prefix= so we know they are surface conditions.

You'll want to drop the column you'll use as the reference category.

Before we do this: which column works best as the reference?

Now build your surf_dummies variable.

Confirm your surf_dummies looks roughly like this:

Another regression#

Let's run another regression to see the impact of both sex and surface condition on fatalities.

Build your train_df#

To build your train_df, I recommend doing it either of these two ways. They both first select the important columns, then add in the one-hot encoded surf_dummies columns.

train_df = pd.DataFrame({
    'col1': merged.col1,
    'col2': merged.col2,
    'col3': merged.col3,
})
train_df = train_df.join(surf_dummies)
train_df = train_df.dropna()

or like this:

train_df = train_df[['col1','col2','col3']].copy()
train_df = train_df.join(surf_dummies)
train_df = train_df.dropna()

The second one is shorter, but the first one makes it easier to use comments to remove columns later.

Run your regression and check your odds ratios#

Actually no, wait, first - what kind of surface do you think will have the highest fatality rate?

Confirm your train_df has 815,843 rows and 9 columns.

• Tip: When you run your regression, if you get an error about not knowing what to do with U, it's because you didn't convert your sex to numbers (or if you did, you didn't do it in your original dataframe)

Is this what you expected? Why do you think this result might be the case?

More features: Vehicles#

Maybe the car they're in is related to the car they were in. Luckily, we have this information - read in combined_vehicle_data as vehicles.

Weights of those cars#

The car weights are stored in another file since the info had to come from an API. I looked up the VINs - vehicle identification numbers - in a government database to try to get data for each of them.

Read them and build a new dataframe that is both the vehicle data along with their weights. You can call it vehicles since you don't need the original weightless vehicle data any more.

Confirm that your combined vehicles dataset should have 534,436 rows and 35 columns. And yes, that's less than we were working with before - you haven't combined it with the people/crashes dataset yet.

Filter your data#

We only want vehicles that are "normal" - somewhere between 1500 and 6000 pounds. Filter your vehicles to only include those in that weight range.

Confirm that you have 532,370 vehicles in the dataset.

Add this vehicle information to your merged data#

Now we'll have a dataframe that contains information on:

• The people themselves and their injuries
• The crash
• The vehicles

Every person came with a VEHICLE_ID column that is the vehicle they were in. You'll want to merge on that.

Confirm you have 99 columns and 616,212 rows. That is a lot of possible features!

Another regression, because we can't get enough#

Build another train_df and run another regression about how car weight impacts the chance of fatalities. You'll want to confirm that your dataset has 616,212 and 2 columns.

Can you translate that into plain English? Remember weight is in pounds.

I feel like pounds isn't the best measure for something like this. Remember how we had to adjust percentages with AP and life expectancy, and then change around the way we said things? It sounded like this:

Every 10% increase in unemployment translates to a year and a half loss of life expectancy

Instead of every single pound, maybe we could do every... some other number of pounds? One hundred? One thousand?

Run another regression with weight in thousands of pounds. Get another odds ratio. Give me another sentence English.

Two-car accidents, struck and striker#

Here's the thing, though: it isn't just the weight of your car. It's the weight of both cars! If I'm in a big car and I have a wreck with a smaller car, it's the smaller car that's in trouble.

To get that value, we need to do some feature engineering, some calculating of new variables from our existing variables.

We need to jump through some hoops to do that.

Two-car accidents#

First we're going to count how many vehicles were in each accident. Since we're looking to compare the weight of two cars hitting each other, we're only going to want crashes with only two cars.

By using .value_counts I can see how many cars were in each crash, and now I'm going to filter to get a list of all of the ones with two vehicles.

And now we'll filter my vehicles so we only have those that were in two-vehicle crashes.

Struck and striker#

To do the math correctly, we need both the risk of someone dying in the smaller car and the risk of someone dying in the bigger car. To do this we need to separate our cars into two groups:

• The 'struck' vehicle: did the person die inside?
• The 'striker' vehicle: how much heavier was it than the struck car?

But we don't know which car was which, so we have to try out both versions - pretending car A was the striker, then pretending car B was the striker. It's hard to explain, but you can read Pounds That Kill - The External Costs of Vehicle Weight.pdf for more details on how it works.

Put people in their cars#

Which car was each person in? We'll assign that now.

Add the crash details#

You did this already! I'm going to do it for you. We're merging on REPORT_NO_x because there are so many REPORT_NO columns duplicated across our files that pandas started giving them weird names.

Filter#

We already filtered out vehicles by weight, so we don't have to do that again.

Calculated features#

I'm sure you forgot what all the features are, so we'll bring back whether there was a fatality or not

Feature: Accident was fatal#

Feature: Weight difference#

Remove everything missing weights for strikers or struck vehicles. You might need to merged.columns to remind yourself what the column names are.

Confirm your dataset has roughly 335,000 rows.

Create a new feature called weight_diff about how much heavier the striking car was compared to the struck car. Make sure you've done the math correctly!

Feature adjustment#

Make all of your weight columns in thousands of pounds instead of just in pounds. It'll help you interpret your results much better.

Another regression!!!#

What is the impact of weight difference on fatality rate? Create your train_df, drop missing values, run your regression, analyze your odds ratios.

Please translate your odds ratio into plain English.

Adding in more features#

How about speed limit? That's important, right? We can add the speed limit of the striking vehicle with SPEED_LIMIT_striker.

Can you translate the speed limit odds ratio into plain English?

Feature engineering: Speed limits#

Honestly, that's a pretty bad way to go about things. What's more fun is if we translate speed limits into bins.

First, we'll use pd.cut to assign each speed limit a category.

Then we'll one-hot encode around 20-30mph speed limits.

Running a regression#

I like this layout for creating train_df, it allows us to easily add dummies and do a little replacing/encoding when we're building binary features like for sex.

If the below gives you an error, it's because SEX_CODE is already a number. In that case, just remove .replace({'M': 1, 'F': 0, 'U': np.nan }).

Describe the impact of the different variables in simple language. What has the largest impact?

Now you pick the features#

Up above you have examples of:

• Creating features from numbers (speed limits)
• Creating features from 0/1 (sex)
• Creating features from binning numbers that are one-hot encoded (speed limit bins - speed_bins)
• Creating features from categories that are one-hot encoded (surface - surf_dummies

What else do you think matters? Try to plug in more features and see if you can get anything interesting.

• Hot tip: The funniest/most interesting thing feature you can add is also the dumbest. Ask me about it in #algorithms if you end up getting down here.