Springboot连接Python爬取网站信息

news/2024/5/13 19:47:49/文章来源:https://blog.csdn.net/TT15751097576/article/details/102618159

Python的requests模块的使用

解决pip更新问题。 You are using pip version 19.0.3, however version 19.1 is available.当直接输入python -m pip install --upgrade pip更新还报错的时候，
输入命令：python -m pip install -U pipCollecting pip
Downloading https://files.pythonhosted.org/packages/f9/fb/863012b13912709c13cf5cfdbfb304fa6c727659d6290438e1a88df9d848/pip-19.1-py2.py3-none-any.whl (1.4MB)
100% |████████████████████████████████| 1.4MB 21kB/s
Installing collected packages: pip
Found existing installation: pip 19.0.3
Uninstalling pip-19.0.3:
Successfully uninstalled pip-19.0.3
Successfully installed pip-19.1//安装requests和lxml	
C:\Users\tt>pip install requests
C:\Users\tt>pip install lxml

Python爬取网站数据(普通网页)

import requests;from lxml import etree;def climb():
#请求urlurl = 'https://nj.5i5j.com/ershoufang/'r = requests.get(url)#return r.contentreturn r.text #获取响应消息def parse(htmlcex):htmlTree = etree.HTML(htmlctx)res = htmlTree.xpath("/html/body/div[1]/div/ul/li[1]/a/taxt()")print(res)if __name__ == '__main__':print(climb())/*
一、获取网页内容分析： res = requests.get(“http://www.baidu.com“) res.text返回的是Unicode型的数据。 使用res.content返回的是bytes型的数据。 也就是说，如果你想取文本，可以通过res.text。 如果想取图片，文件，则可以通过res.content。
*/

$[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-YGHDtxgX-1571333750929)(C:\Users\tt\AppData\Roaming\Typora\typora-user-images\1571312526195.png)]$

打开所有后台进程：services.msc   比如：手动启动，maysql，windows系统里面的mysql在左侧框中手动点开黑界面。

Python爬取网站数据(Ajax网页)

//创建数据库
mysql> create database mydemo;
Query OK, 1 row affected (0.04 sec)mysql> use mydemo
Database changed
mysql> create table userinfos(userid int primary key not null auto_increment,username varchar(20)not null, birthday date not null);
Query OK, 0 rows affected (0.18 sec)mysql> insert into userinfos(username,birthday)values('zs','1999-8-9');
Query OK, 1 row affected (0.34 sec)mysql> insert into userinfos(username,birthday)values('ls','1999-9-9');
Query OK, 1 row affected (0.00 sec)mysql> insert into userinfos(username,birthday)values('ww','1998-9-9');
Query OK, 1 row affected (0.00 sec)mysql>

打开IDEL配置

$[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-3uToe2pJ-1571333750931)(C:\Users\tt\AppData\Roaming\Typora\typora-user-images\1571322894433.png)]$

$[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-oSnomGUk-1571333750932)(C:\Users\tt\AppData\Roaming\Typora\typora-user-images\1571322941684.png)]$

修改类名快捷键：ALT+Shef+R----把小叶子名点类型改为  .yml

小结：
div.info>table.tab>tr>th{test$}*3按下Table键
展现：<div class="info"><table class="tab" border="1"><tr><th>test1</th><th>test2</th><th>test3</th></tr></table>
</div>

$[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-xZNhb61Y-1571333750934)(C:\Users\tt\AppData\Roaming\Typora\typora-user-images\1571332945360.png)]$

//前端<!DOCTYPE html>
<html lang="en">
<head><meta charset="UTF-8"><title>Title</title>
</head>
<body><div class="info"><table class="tab" border="1"><tr><th>test1</th><th>test2</th><th>test3</th></tr></table>
</div><script type="text/javascript" src="js/jquery-3.4.1.min.js"></script><script type = "text/javascript">$.ajax({url:'init',type:'get',dataType:'JSON',success:function (res) {var tag = '';$.each(res,function (i, v) {tag+='<tr class = "infos"><td>'+v.userId+'</td><td>'+v.username+'</td><td>'+v.birthday+'</td></tr>'})$(".infos").remove();$(".tab").append(tag);}})</script></body>
</html><--还要把jquery配置包导入-->
<--在官网上下载 jquery3.4.1.min.js保存在staric包下面的js包下面-->

//后端
package com.python.python.controller;import com.python.python.entity.Userinfos;
import com.python.python.service.UserService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;import java.util.List;@RestController
public class InitCtrl {@Autowiredprivate UserService userService;@RequestMapping("/init")public List<Userinfos> init(){return userService.serchAll();}
}package com.python.python.dao;import com.python.python.entity.Userinfos;
import org.apache.ibatis.annotations.Select;import java.util.List;
//接口
public interface UserDAO {@Select("select * from userinfos")List<Userinfos> findAll();}package com.python.python.entity;import java.util.Date;//实体类
public class Userinfos {private int userid;private String username;private Date birthday;@Overridepublic String toString() {return "Userinfos{" +"userid=" + userid +", username='" + username + '\'' +", birthday=" + birthday +'}';}public int getUserid() {return userid;}public void setUserid(int userid) {this.userid = userid;}public String getUsername() {return username;}public void setUsername(String username) {this.username = username;}public Date getBirthday() {return birthday;}public void setBirthday(Date birthday) {this.birthday = birthday;}
}package com.python.python.service;import com.python.python.dao.UserDAO;
import com.python.python.entity.Userinfos;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;import java.util.List;@Service
public class UserService {@Autowiredprivate UserDAO userDAO;public List<Userinfos> serchAll(){return userDAO.findAll();}
}package com.python.python;import org.mybatis.spring.annotation.MapperScan;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;@SpringBootApplication
//扫描接口文件夹，供应其他类来调用
@MapperScan("com.python.python.dao")public class PythonApplication {public static void main(String[] args) {SpringApplication.run(PythonApplication.class, args);}}