计算机毕业设计Hadoop+Spark+Hive租房推荐系统贝壳租房数据分析租房爬虫租房可视化租房大数据大数据毕业设计大数据毕设机器学习

毕业技术方向调查表

姓名：李昌福

课题方向	房无忧房屋租赁平台
开发语言：	Java	前端框架：		VUE		数据库：	MySQL
服务器端框架：	SpringCloud		其他技术：		Hadoop、HDFS
方向意义	结合四年在校所学专业知识，针对如今人们对住房需求提升的问题，进行调研和分析，并利用Java、VUE、SpringCloud等技术开发XX房屋租赁平台，解决人们找房难、出租难的问题，并提供数据分析结果便于用户对房源及租期进行合理的规划。
预设业务逻辑	模块一：租客用户模块功能点1：用户注册和登录功能点2：查看在租房屋的具体房源信息（时间，所在位置，大小等）功能点3：向房东提出看房请求功能点4：对已租房屋向房东发起退租请求功能点5：查看租房历史，并可对其进行增删改查模块二：房东用户模块功能点1：用户注册和登录功能点2：发布房源具体信息（包括图片、文字、视频等）功能点3：查阅看房请求（所对应的租客信息、时间、请求的房源）功能点4：管理看房请求（可对其接受或拒绝）和退租请求模块三：管理员模块功能点1：管理员注册和登录功能点2：查看平台租客、房东权限和信息，并可对其进行管理功能点3：查看平台的房源内容，并有权限对其进行增删改查功能点4：发布平台公告，返回公告已确认信息模块四：报障模块功能点1：租客发现故障，进行报障申请功能点2：房东查看对应租客未处理的故障功能点3：房东收到报障申请，开始处理已报故障功能点4：发布故障处理流程和处理结果反馈模块五：数据导出及分析模块功能点1：将用户数据导出为MR平台数据文件功能点2：允许用户基于HDFS分布式平台进行数据管理功能点3：允许用户基于Hadoop集群进行数据处理功能点4：处理数据，得出看房请求的最大值，请求量与时间的关系功能点5：处理并分析租客年龄信息，对比各年龄人群租房的偏好功能点6：输出数据成为数据库文件，供数据展示平台使用
技术或业务逻辑特色

核心算法代码分享如下：

from selenium import webdriverfrom selenium.webdriver.common.by import By
from lxml import etree
import time
from selenium.webdriver.chrome.options import Options
import pymysql
import re
import json#一线城市租房信息
#cities = ['bj','sh','gz','sz']
cities = ['sz']
options = Options()
driver = webdriver.Chrome(executable_path=r'chromedriver.exe', options=options)def get_url_info(url):driver.get(url)#driver.set_page_load_timeout(60)time.sleep(40)driver.refresh()driver.minimize_window()zufang = driver.find_element(By.XPATH,'/html/body/div[3]/div[1]/div[1]/div/div[1]/div[1]/span[1]/a')zufang.click()#driver.close()driver.switch_to.window(driver.window_handles[-1])time.sleep(1)page_source = driver.page_source;hs = etree.HTML(page_source)nums = driver.find_element(By.XPATH, '//*[@id="pager_wrap"]/div[@class="pager"]/a[@class="next"]/preceding-sibling::a[1]/span').textif nums and len(nums)>0:end = int(nums)else:end = 20;for j in range(0,end+1):ep_logs = hs.xpath('//ul[@class="house-list"]/li/@ep-log')imgs = hs.xpath('//ul[@class="house-list"]/li[@ep-log]/div[@class="img-list"]/a/img/@src')urls = hs.xpath('//ul[@class="house-list"]/li[@ep-log]/div[@class="des"]/h2/a/@href')decs = hs.xpath('//ul[@class="house-list"]/li[@ep-log]/div[@class="des"]/h2/a/text()')prices = hs.xpath('//ul[@class="house-list"]/li[@ep-log]/div[@class="list-li-right"]/div[@class="money"]/b/text()')danweis = hs.xpath('//ul[@class="house-list"]/li[@ep-log]/div[@class="list-li-right"]/div[@class="money"]/b/following-sibling::text()')next = driver.find_element(By.XPATH,'//*[@id="pager_wrap"]/div[@class="pager"]/a[@class="next"]')length = len(ep_logs)for i in range(0,length):data = {}dec = decs[i].split('|')json_obj = json.loads(ep_logs[i])print( json_obj['houseid'] )#data['id'] = json_obj['houseid'] #房间ID#if not_exists(houseid=data['id']):if not_exists(houseid= json_obj['houseid'] ):data['pic'] = imgs[i] #房间图片链接data['url'] = urls[i] #房间URL链接data['house_title'] = dec[1].strip() #房间标题data['rent_way'] = dec[0].strip() #租房模式data['house_pay'] = ''.join([prices[i],danweis[i].strip()]) #价格time.sleep(3)driver.get(data['url']) #进入详情页ps_inner = driver.page_sourcehs_inner = etree.HTML(ps_inner)pay_way = hs_inner.xpath('//span[@class="instructions"]/text()')#pay_way = hs_inner.xpath('/html/body/div[3]/div[2]/div[2]/div[1]/div[1]/div/span[2]/text()')#type_str = hs_inner.xpath('/html/body/div[3]/div[2]/div[2]/div[1]/div[1]/ul/li[2]/span[2]/text()')type_str = hs_inner.xpath('//ul[@class="f14"]/li[2]/span[2]/text()')#floor_str = hs_inner.xpath('/html/body/div[3]/div[2]/div[2]/div[1]/div[1]/ul/li[3]/span[2]/text()')floor_str = hs_inner.xpath('//ul[@class="f14"]/li[3]/span[2]/text()')# estate = hs_inner.xpath('/html/body/div[3]/div[2]/div[2]/div[1]/div[1]/ul/li[4]/span[2]/a/text()')estate = hs_inner.xpath('//ul[@class="f14"]/li[4]/span[2]/a/text()')# areas = hs_inner.xpath('/html/body/div[3]/div[2]/div[2]/div[1]/div[1]/ul/li[5]/span[2]/a[1]/text()')areas = hs_inner.xpath('//ul[@class="f14"]/li[5]/span[2]/a/text()')addresses = hs_inner.xpath('//span[@class="dz"]/text()')#addresses = hs_inner.xpath('/html/body/div[3]/div[2]/div[2]/div[1]/div[1]/ul/li[6]/span[2]/text()')#times = hs_inner.xpath('/html/body/div[3]/div[1]/p/text()')times = hs_inner.xpath('//div[@class="house-title"]/p/text()')agents = hs_inner.xpath('//*[@id="vipAgent"]/div[1]/p[1]/a/text()')disposals = hs_inner.xpath('//ul[@class="house-disposal"]/li[not(@class="no-config")]/text()')spots = hs_inner.xpath('//ul[@class="introduce-item"]/li[1]/span[2]/em/text()')#descs = hs_inner.xpath('//ul[@class="introduce-item"]/li[2]/span[2]/em/text()')descs = hs_inner.xpath('//ul[@class="introduce-item"]//li[3]/span[2]/text()')print(descs)if pay_way and len(pay_way)>0:data['house_pay_way'] = pay_way[0]if type_str and len(type_str)>0:type_str = type_str[0]types = type_str.split("\xa0\xa0")if types and len(types)==3:data['house_type'] = types[0]data['house_area'] = types[1].split(' ')[0]+"平"data['house_decora'] = types[2]elif types and len(types)==2:data['house_type'] = types[0]data['house_area'] = types[1]elif types and len(types)==1:data['house_type'] = types[0]if floor_str and len(floor_str)>0:floor_str = floor_str[0]floors = floor_str.split("\xa0\xa0")if floors and len(floors) == 2:data['toward'] = floors[0]f = floors[1].split('/')if f and len(f) == 2:data['floor'] = f[0]data['floor_height'] = f[1]elif f and len(f) == 1:data['floor'] = re.findall('\d{1,2}',f[0])[0]+'层'elif floors and len(floors) == 1:data['toward'] = floors[0]if estate and len(estate)>0:data['house_estate'] = estate[0].strip()if areas and len(areas)>0:data['area'] = areas[0]if addresses and len(addresses)>0:data['address'] = addresses[0].strip()if times and len(times)>0:times = times[len(times)-1].strip()data['time'] = times.split('\xa0')[0]print("时间："+data['time'])if agents and len(agents)>0:data['agent_name'] = agents[0].strip()if disposals and len(disposals)>0:data['house_disposal'] = ' '.join(disposals).strip()if spots and len(spots)>0:data['house_spot'] = ' '.join(spots)if descs and len(descs)>0:data['house_desc'] = descs[0]print(data)to_mysql(data)driver.back()time.sleep(1)next.click()def not_exists(houseid):"""信息写入mysql"""table = 'house_info'db = pymysql.connect(host='localhost', user='root', password='123456', port=3396, db='model')cursor = db.cursor()sql_search = "SELECT COUNT(1) FROM {table} where id = {id}".format(table=table,id=houseid)cursor.execute(sql_search)data_sql = cursor.fetchall()count = data_sql[0][0]if count > 0:print('exists')return Falsereturn Truedef to_mysql(data):"""信息写入mysql"""table = 'house_info'keys = ', '.join(data.keys())values = ', '.join(['%s'] * len(data))db = pymysql.connect(host='localhost', user='root', password='123456', port=3396, db='model')cursor = db.cursor()sql = 'INSERT INTO {table}({keys}) VALUES ({values})'.format(table=table, keys=keys, values=values)try:datas = data.values()if cursor.execute(sql, tuple(datas)):print("Successful")db.commit()except:print('Failed')db.rollback()db.close()if __name__ == '__main__':for i in cities:url = 'XXXXX'%iget_url_info(url)