Introduction: Business Problem
This research is meant to solve the problems faced by employees who are being transferred to the city of Chennai in large numbers due to relocation of various IT companies there. In the study we leverage Foursquare location data to find patterns and clusters of restaurants to make the lives easier of those employees that do volunteer to be transferred to the city. As food is a major factor that one has to consider when moving to a new city, this study focuses to finding out if there are ample number of restaurants in the area so that the employees can live in those areas that have abundant restaurants. Also as setting up offices in Chennai is very economical for companies, this seems to be a desired solution for companies wishing to make large budgetary savings on cutting operational costs by moving operations to the city of Chennai.
In this study Foursquare has been used to obtain location data to solve the problem of finding neighborhoods in Chennai that have a large number of food outlets and restaurants. Firstly, the number of neighborhoods and their respective coordinates were to be retrieved, so that through Foursquare it was possible to find nearby venues. Using the data, Foursquare should display search results for nearby venues and get their respective categories.
Once the venues were located they were then clustered using k-means. The cluster in which eating places are of the highest frequency will be the the group of neighborhoods that we are looking for. All of these neighborhoods are likely to be suitable for the employees to move in. This method can also be used to find out other requirements of the employees that move. For example we could use this to find out which areas have malls, shops, good transport access, low flat prices, etc. The target audience here are employees who are moving to a new city and who require some information about the neighborhoods prior to deciding what area to select when they relocate to Chennai
Data for Analysis
Since there was not much data on the city of Chennai online, The list of neighborhoods were manually scraped from the Wikipedia page https://en.wikipedia.org/wiki/Areas_of_Chennai using bs4, and then all the hyperlinks were grabbed. Using urllib, these links are visited individually and the coordinates and pincodes are scraped and entered into a pandas dataframe.
url = 'https://en.wikipedia.org/wiki/Areas_of_Chennai'
page_unparsed = urllib.request.urlopen(url)
soup = BeautifulSoup(page_unparsed, 'html.parser')
wiki_rows = [] # each row in the wikipedia table
urls = []
names = []
wiki_table = soup.find_all("table", {"class": "wikitable"})
for row in wiki_table:
wiki_rows.append(row.find_all('a', href=True))
# gets names and links of each neighborhood so that further scraping can be done
for i in range(len(wiki_rows[0])):
urls.append('https://en.wikipedia.org' + wiki_rows[0][i]['href'])
names.append(wiki_rows[0][i].text)
# getting data from each neighborhood
latitudes = []
longitudes = []
pincodes = []
for url in tqdm_notebook(urls, total = len(urls), unit = 'url'):
try: # because some links are broken
page_unparsed = urllib.request.urlopen(url)
soup = BeautifulSoup(page_unparsed, 'html.parser')
except:
continue
coords = soup.find("span", {"class" : "geo-dec"})
pincode = soup.find("div", {"class" : "postal-code"})
if coords == None: # because some pages do not have coordinates listed
latitudes.append(np.nan)
longitudes.append(np.nan)
else:
coords = coords.text.split()
latitudes.append(float(coords[0].replace('N', '').replace('°', '')))
longitudes.append(float(coords[1].replace('E', '').replace('°', '')))
neighborhoods = pd.DataFrame(list(zip(names, latitudes, longitudes)), columns =['Name', 'Latitude', 'Longitude'])
neighborhoods = neighborhoods[neighborhoods['Latitude'].notnull()]
neighborhoods = neighborhoods[neighborhoods['Longitude'].notnull()]
neighborhoods.head()
Foursquare
The next step is to get all the venues in each neighborhood within a specified radius, in this case, 500 meters was selected. To use Foursquare, we need to create an account as they offer a limited number of API calls per day for a free user (more if you give your credit card details). After signing up for a developer accout, we will get a CLIENT_ID and a CLIENT_SECRET which are then appended to a URL along with other parameters, which is then used to send a GET request to that URL. The resulting JSON file is then parsed and stored for further use.
url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
CLIENT_ID[::-1],
CLIENT_SECRET[::-1],
VERSION,
lat,
lng,
radius,
LIMIT)
# make the GET request
results = requests.get(url).json()["response"]['groups'][0]['items']
Methodology
After this, the venue categories are one-hot encoded and then the 10 most frequent venues (only five selected) in each neighborhood are found.
The result is this: | | Neighborhood | 1st Most Common Venue | 2nd Most Common Venue | 3rd Most Common Venue | 4th Most Common Venue | 5th Most Common Venue |
After this, k-means clustering is used to group these neighborhoods. After they are clustered, we can use Folium (which is a map visualization library for Python), to see all the neighborhoods and the cluster they belong to on a map:
chennai_merged = chennai_merged[chennai_merged['Cluster Labels'].notnull()]
# create map
map_clusters = folium.Map(location=[13.067439, 80.237617], zoom_start=11)
colors = ["#ff0000", "#3d84ad", "#000000", "#ffff00"]
# add markers to the map
markers_colors = []
for lat, lon, poi, cluster in zip(chennai_merged['Latitude'], chennai_merged['Longitude'], chennai_merged['Name'], chennai_merged['Cluster Labels']):
label = folium.Popup(str(poi) + ' Cluster ' + str(cluster), parse_html=True)
folium.CircleMarker(
[lat, lon],
radius=5,
popup=label,
color=colors[int(cluster)],
fill=True,
fill_color=colors[int(cluster)],
fill_opacity=0.7).add_to(map_clusters)
map_clusters
From the map we can see that the clusters are grouped together not due to their Euclidean distances on the map, but due to properties of the venues themselves. Inspecting the dataframe which has the cluster labels along with the most frequent venue in each neighborhood, we can see that:
- Cluster 0 (Pink) have Indian Restaurants as their most frequent venue.
- Cluster 1 (Green) represents various food venues
- Cluster 2 (Black) has pharmacies as the most frequent venues.
- Cluster 3 (Yellow) has multiple forms of transport as the most frequent venues
From the above graph we can see the most frequent venues of cluster 0
Using these results, we can solve our problem. Employees should move to neighborhoods in either Cluster 0 or Cluster 1, as they have a high concentration of restaurants and other food-related venues.
Results
We may have solved our problem and have managed to find Cluster 0 and Cluster 1 as the most suitable areas of relocation for the employees. But as Foursquare data for the city of Chennai is considerably sparse compared to other well developed cities. It also probably does not take into consideration the thousands of small food shops scattered throughout the city
Discussion
Employees would not have trouble finding food in the city of Chennai. For lesser travel times, one can choose any of the neighborhoods in Cluster 0, however one might find that most of them are Indian restaurants. Independent of travel distance, the cluster choice does not matter much as there are more restaurants than any other venue. There is simply not enough data to do an in-depth analysis. The outcome of the project was impacted by the limited effectiveness of the Foursquare API for a city like Chennai. Using some other more developed city might have probably yielded better results. The way the neighborhoods were suggested were majorly due to Indian restaurants alone. Further analysis could’ve been done by using the rating of each venue however yet again ratings were not available for venues in Chennai. If ratings were available, individual restaurants in each locality could’ve been suggested.
Conclusion
Thus through this analysis we can concluded that as cluster 0 and cluster 1 represent the venues that have the most restaurants, it is more likely that the employees would find it more suitable to live in these areas. To get a clearer picture of the senario more data would need to be obtained to have a more in-depth analysis of the scenario.
No comments:
Post a Comment