The Wayback Machine - https://web.archive.org/web/20200624143653/https://github.com/ideawu/ssdb/issues/1179
Skip to content

/ ssdb

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

持续写入数据后出现异常重连，出现fd: 816, link parse error, delete link日志记录 #1179

Open
hongxiaolin opened this issue Dec 26, 2017 · 7 comments
Open

持续写入数据后出现异常重连，出现fd: 816, link parse error, delete link日志记录 #1179

hongxiaolin opened this issue Dec 26, 2017 · 7 comments

Comments

@hongxiaolin
Copy link

@hongxiaolin hongxiaolin commented Dec 26, 2017

持续写入数据，估计是硬盘写入跟不上，跑出异常，程序自动重新建立连接，一直建立不了新连接。查看SSDB日志一直出现下面的记录，这是什么问题？

2017-12-26 23:31:59.579 [WARN ] fd: 816, link parse error, delete link
2017-12-26 23:32:00.163 [DEBUG] new link from 127.0.0.1:65100, fd: 1280, links: 2
2017-12-26 23:32:01.501 [WARN ] fd: 1280, link parse error, delete link
2017-12-26 23:32:04.900 [DEBUG] new link from 127.0.0.1:65102, fd: 864, links: 2
2017-12-26 23:32:06.038 [WARN ] fd: 864, link parse error, delete link
2017-12-26 23:32:07.071 [DEBUG] new link from 127.0.0.1:65103, fd: 1268, links: 2
2017-12-26 23:32:08.296 [WARN ] fd: 1268, link parse error, delete link
2017-12-26 23:32:09.119 [DEBUG] new link from 127.0.0.1:65104, fd: 1028, links: 2
2017-12-26 23:32:10.734 [WARN ] fd: 1028, link parse error, delete link
2017-12-26 23:32:11.192 [DEBUG] new link from 127.0.0.1:65105, fd: 780, links: 2
2017-12-26 23:32:13.256 [WARN ] fd: 780, link parse error, delete link
2017-12-26 23:32:13.794 [DEBUG] new link from 127.0.0.1:65111, fd: 784, links: 2
2017-12-26 23:32:15.550 [WARN ] fd: 784, link parse error, delete link
2017-12-26 23:32:16.532 [DEBUG] new link from 127.0.0.1:65114, fd: 1272, links: 2
2017-12-26 23:32:18.205 [WARN ] fd: 1272, link parse error, delete link
2017-12-26 23:32:18.707 [DEBUG] new link from 127.0.0.1:65115, fd: 768, links: 2
2017-12-26 23:32:20.343 [WARN ] fd: 768, link parse error, delete link
2017-12-26 23:32:21.416 [DEBUG] new link from 127.0.0.1:65116, fd: 1312, links: 2
2017-12-26 23:32:22.913 [WARN ] fd: 1312, link parse error, delete link
2017-12-26 23:32:23.447 [DEBUG] new link from 127.0.0.1:65118, fd: 1016, links: 2
2017-12-26 23:32:25.076 [WARN ] fd: 1016, link parse error, delete link
2017-12-26 23:32:31.227 [DEBUG] new link from 127.0.0.1:65120, fd: 1000, links: 2
2017-12-26 23:32:32.550 [WARN ] fd: 1000, link parse error, delete link
2017-12-26 23:32:33.127 [DEBUG] new link from 127.0.0.1:65124, fd: 1000, links: 2
2017-12-26 23:32:34.602 [WARN ] fd: 1000, link parse error, delete link
2017-12-26 23:32:35.342 [DEBUG] new link from 127.0.0.1:65127, fd: 832, links: 2
2017-12-26 23:32:36.700 [WARN ] fd: 832, link parse erro
@ideawu
Copy link
Owner

@ideawu ideawu commented Dec 27, 2017

应该是你错误地使用了多线程。如果无法正确地使用多线程，建议每个线程建立自己的连接。
@hongxiaolin
Copy link
Author

@hongxiaolin hongxiaolin commented Dec 28, 2017

我是在每个线程里面创建了一个SSDB的连接，构造类里面创建：
private class CommitThread extends Thread{
private SSDB ssdb;
private IQueue pointDataQueue;
private int lastCommitCount = 0;

	public CommitThread(IQueue<HZPointData> pdq){
		try {
			ssdb = new SSDB(rdbHost, rdbPort, timeout);
		} catch (Exception e) {
			logger.error(e);
		}
		this.pointDataQueue = pdq;
	}

在持续不断地写入数据，但是如果数据量大于15万条/秒kv后，就会出现time out（我的timeout设置已经是100000了）。在出现异常后，我会尝试关闭ssdb.close，然后再次连接，写入数据。
try{
ssdb.multi_set(keyvalues);
logger.info("提交队列"+pointDataQueue.getName()+"缓存数据 " + keyvalues.length/2 + " 条，用时:" + (System.currentTimeMillis() - s) + "ms");
}catch(Exception e){
//关闭连接
try {
ssdb.close();
//rdb = null;
} catch (Exception e1) {logger.error(e1);}

			//出现异常后，尝试重新连接再次写入
			logger.error("队列"+pointDataQueue.getName()+"写入异常:" + e.getCause() + " 重新连接，尝试再次写入。");
			//重连接
			try {
				ssdb = new SSDB(rdbHost, rdbPort, timeout);
			} catch (Exception e2) {
				logger.error("再次连接失败：" + e2);
			}
			try {
				ssdb.multi_set(keyvalues);
			} catch (Exception e2) {
				logger.error("再次写入失败：" + e2);
			}

但是再次写入就基本上写不进去了。

SSDB日志就会报上面的错误信息。
@tiankonguse
Copy link

@tiankonguse tiankonguse commented Jan 7, 2018

高并发性确实可能出现这种情况。
Link::read 函数收包时，服务收的数据可能不完整
然后在RedisLink::parse_req 解析时，对于不完整的包可能会返回-1，从而返回 link parse error, delete link 错误。
@hongxiaolin
Copy link
Author

@hongxiaolin hongxiaolin commented Jan 7, 2018

谢谢！@tiankonguse 这个问题如何能够避免？每次写入multi_set的时候包小点，会不会好点？
@tiankonguse
Copy link

@tiankonguse tiankonguse commented Jan 8, 2018

每秒15W, 写什么这么大的量？ 压测吗？
@hongxiaolin
Copy link
Author

@hongxiaolin hongxiaolin commented Jan 8, 2018

不是压力测试，是真实的生产环境。每秒采集15万个采集点的实时数据，并且要写入到库中。
@tiankonguse
Copy link

@tiankonguse tiankonguse commented Jan 10, 2018

ssdb的机器只有一台吗？ 这样的话太大了，建议加一个队列
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
3 participants
@tiankonguse @ideawu @hongxiaolin
You can’t perform that action at this time.